From perception to action: the VLA idea

"Every other model in this book stops when it has an answer. I only start there: my answer has to move a real gripper, and the gripper does not care how confident I feel."

An Embodied AI Agent

What breaks in the classical pipeline

The textbook robotics stack is a chain: perception detects objects, a state estimator fuses them into a world model, a planner computes a collision-free trajectory, and a controller tracks it. Each stage has clean inputs and outputs, and each has been engineered for decades. The trouble is the seams. Perception hands the planner a symbolic scene ("mug at pose X"), and the moment the real world does not fit the schema (the mug is transparent, half-occluded, or simply an object nobody enumerated) the abstraction leaks and the downstream stages act confidently on a wrong premise. Worse, the instruction "clear the table" has no place to enter this pipeline: it is neither a pose nor a trajectory, and translating open-ended human intent into planner goals is exactly the part no one automated well.

A VLA collapses the chain. It treats robot control as a supervised learning problem: given an observation \(o_t\) (camera images plus proprioceptive state) and a language instruction \(\ell\), predict the action \(a_t\) a competent demonstrator would take. Formally it learns a policy

trained by behavior cloning to match a dataset of teleoperated demonstrations. There is no explicit object detector, no symbolic planner, no hand-tuned interface between stages to leak. The instruction enters naturally, because the same network that reads the pixels also reads the words. What replaces the engineered pipeline is data: the model must see enough demonstrations to learn the perception-to-action mapping that engineers used to build by hand.

What "action" means as a model output

A classifier outputs a category; a VLA outputs a command a controller can execute. Concretely, an action \(a_t\) is a short vector: most commonly a change in end-effector pose (three translations, three rotations) plus a gripper open/close signal, expressed as a delta to apply over the next control step. Some policies output absolute target poses, others output raw joint-position targets; the interface is a design choice, but the shape is always "a handful of continuous numbers, many times per second."

Two ideas make this learnable with transformer machinery. The first is action tokenization: each continuous action dimension is discretized into a fixed number of bins, so predicting an action becomes predicting a short sequence of tokens, exactly the task language models already excel at. The policy emits tokens; a detokenizer maps them back to metric commands. The second is action chunking: rather than predict one step and re-plan, the policy predicts a short horizon of \(H\) future actions at once and executes them open-loop before querying again. Chunking cuts the number of expensive forward passes, and, more importantly, it suppresses the compounding, jittery errors that a step-by-step policy accumulates, because a committed multi-step plan is smoother than a sequence of independently sampled twitches. The cost is reactivity: a long chunk cannot respond to a surprise mid-execution, so \(H\) trades smoothness against responsiveness.

import numpy as np

# A VLA's action space for a 6-DoF arm + gripper: 7 continuous dims.
# Each dim is discretized into 256 bins over a known metric range.
ACTION_LOW  = np.array([-0.05, -0.05, -0.05, -0.10, -0.10, -0.10, 0.0])  # dx,dy,dz,droll,dpitch,dyaw,grip
ACTION_HIGH = np.array([ 0.05,  0.05,  0.05,  0.10,  0.10,  0.10, 1.0])
N_BINS = 256

def tokenize(action):
    """Continuous action -> integer tokens the transformer predicts."""
    clipped = np.clip(action, ACTION_LOW, ACTION_HIGH)
    frac = (clipped - ACTION_LOW) / (ACTION_HIGH - ACTION_LOW)
    return np.round(frac * (N_BINS - 1)).astype(int)

def detokenize(tokens):
    """Predicted tokens -> metric command the controller executes."""
    frac = tokens / (N_BINS - 1)
    return ACTION_LOW + frac * (ACTION_HIGH - ACTION_LOW)

# One action chunk = H future actions predicted in a single forward pass.
H = 8
demo_chunk = np.random.uniform(ACTION_LOW, ACTION_HIGH, size=(H, 7))
tokens = np.stack([tokenize(a) for a in demo_chunk])   # what the model learns to emit
recovered = np.stack([detokenize(t) for t in tokens])  # what the robot actually runs

err = np.abs(recovered - demo_chunk).max()
print(f"chunk shape (H x dims): {tokens.shape}")
print(f"max detokenization error: {err:.4f}  (bounded by one bin width)")

The snippet above is deliberately minimal, but it exposes the load-bearing choices. The bin count sets the finest motion the policy can command (256 bins over a 10 cm range resolves under half a millimeter per step), and the metric range must bracket the demonstrations or the clipping silently caps real motions. Predicting the chunk as a token grid is precisely what lets a vision-language backbone, designed to emit text, instead emit robot commands with no change to its architecture.

The perception-to-action loop on real hardware

A VLA is not run once; it is run in a loop against a moving world. At each cycle the robot captures its observation (typically one or more RGB views plus the proprioceptive state from Chapter 57), the policy conditions on that observation and the fixed instruction, predicts an action chunk, and a low-level controller executes the chunk while the next observation is already being gathered. Two clocks matter here and they differ by an order of magnitude. The policy runs at perhaps 3 to 10 Hz because a large transformer forward pass is expensive, while the joint controller runs at hundreds of hertz to keep the arm stable. Action chunking bridges the gap: one slow policy query supplies many fast controller setpoints. This is the same fast-backbone, slow-update pattern that legged state estimation used in Chapter 57, turned around to drive action instead of estimate state.

A subtlety separates a VLA from a plain image classifier: it is a closed-loop system, so its own errors feed back into its future inputs. If the policy nudges the gripper slightly off course, the next observation shows that off-course scene, which was rare in the demonstrations, so the next prediction can be worse still. This covariate shift is the classic failure mode of behavior cloning, and it is why demonstration coverage, recovery examples, and action chunking (which limits how far a single mistake propagates before the next correction) all matter more here than in any offline perception task. It also connects the VLA to the predictive world-model view of Chapter 53: acting well requires an implicit sense of how the scene will respond to the action, even when the model never predicts a future frame explicitly.

Why fold it all into one model

The unifying argument for VLAs has three strands. Generalization: a modular pipeline generalizes only as far as its weakest hand-built stage, whereas a single network trained on diverse data can transfer visual and linguistic knowledge across the seams that used to break, recognizing a novel object it was never explicitly programmed to detect. Instruction-following: because language and control share parameters, open-ended human intent enters as an input rather than requiring a bespoke goal encoder, which is the capability classical robotics most lacked. Scaling: the approach improves with more demonstrations and larger backbones the way language models improve with more text, so progress becomes a data-and-compute problem rather than an endless-engineering problem. The counterweights are real and worth stating plainly: demonstrations are costly, closed-loop behavior cloning suffers covariate shift, a black-box policy is hard to certify for safety, and the calibrated uncertainty machinery of Chapter 18 is far less mature for actions than for predictions. The rest of this chapter is the story of how the field is pushing each strand forward and blunting each counterweight.

What's Next

In Section 58.2, we make the transfer argument concrete: the RT-1 and RT-2 models show how a policy built on a web-pretrained vision-language backbone inherits semantic knowledge it never saw a robot demonstrate, letting it act on objects and instructions that appear in no teleoperation log. That is the first payoff of folding perception and action into one model, and the reason the rest of the chapter builds on VLAs rather than pipelines.