RT-1/RT-2 and web-scale knowledge transfer

"They asked me to pick up the extinct animal. Nobody ever showed me a dinosaur on a tabletop. I had, however, read the entire internet, so I reached for the plastic T-rex."

A Suspiciously Well-Read AI Agent

This section assumes the transformer machinery of Chapter 15 (tokens, attention, next-token prediction) and the vision-language models of Chapter 21, which RT-2 reuses wholesale as its backbone. Both models come from Google's robotics effort; RT stands for Robotics Transformer.

RT-1: one transformer that eats a whole fleet's data

RT-1 (2022) answered a narrower question than RT-2: can a single network absorb large, diverse, real robot data and stay reliable? Prior policies were trained per task and brittle. RT-1 was trained on roughly 130,000 teleoperated demonstrations spanning about 700 language-described tasks (pick, place, open drawer, knock over, move near), collected over months on a fleet of thirteen mobile manipulators. The architecture is deliberately plain: each camera frame passes through an ImageNet-pretrained EfficientNet whose features are conditioned on the instruction by FiLM layers (the language embedding scales and shifts the vision features, so "pick up the apple" and "pick up the can" light up different channels), a TokenLearner module compresses each frame to eight learned tokens to keep the sequence short, and a decoder-only transformer attends over a short history of frames and emits the action for the current step. The whole thing runs closed-loop at about 3 Hz on the robot.

The decision that made RT-1 matter, though, is not the vision trunk. It is how the output is represented.

Actions as tokens: casting control as next-token prediction

A manipulator command at one step is a small vector: a change in end-effector position \((\Delta x, \Delta y, \Delta z)\), a change in orientation (three more numbers), a gripper open/close command, and a discrete mode flag that says whether the arm is moving, the base is moving, or the episode should terminate. RT-1 turns each continuous dimension into a categorical variable by discretizing it into \(K = 256\) uniform bins. For dimension \(j\) with range \([a_j^{\min}, a_j^{\max}]\), the action value \(a_j\) becomes the integer bin

Now an action is just a short string of integer tokens, and predicting it is exactly the categorical next-token problem transformers are built for: the policy outputs a softmax over 256 classes per dimension and is trained with plain cross-entropy, no specialized regression head or Gaussian-mixture output. At run time you take the arg-max bin per dimension and invert the map to get a real command. The price is quantization error bounded by half a bin width, which the code below measures.

import numpy as np

K = 256  # bins per action dimension, as in RT-1 / RT-2
# One 7-DoF end-effector action: [dx, dy, dz, droll, dpitch, dyaw, gripper]
lo = np.array([-0.05, -0.05, -0.05, -0.25, -0.25, -0.25, 0.0])
hi = np.array([ 0.05,  0.05,  0.05,  0.25,  0.25,  0.25, 1.0])

def to_tokens(a):
    frac = (a - lo) / (hi - lo)
    return np.clip(np.round(frac * (K - 1)), 0, K - 1).astype(int)

def from_tokens(b):
    return lo + (b / (K - 1)) * (hi - lo)

a = np.array([0.021, -0.034, 0.005, 0.11, -0.19, 0.02, 1.0])
b = to_tokens(a)                       # -> the action as 7 integer tokens
a_hat = from_tokens(b)                 # decode back to a command
print("tokens      :", b)
print("max abs err :", np.max(np.abs(a - a_hat)))   # <= half a bin width

The snippet is the whole conceptual bridge. Once actions are tokens over a fixed vocabulary, a policy is a language model whose sentences happen to be motions, and any architecture that predicts tokens can drive a robot. RT-1 built that language model from scratch on robot data alone. RT-2 asked the obvious follow-up: why start from scratch when a model that already speaks token has read the internet?

RT-2: co-training on the web so semantics transfer

RT-2 (2023) takes a large pretrained vision-language model, either PaLI-X or PaLM-E with billions of parameters, and co-fine-tunes it on a mixture of two data sources at once: the original web vision-language tasks (visual question answering, captioning, grounding) and the RT-1 robot demonstrations, with the robot actions written in the token scheme above. To make actions share the model's existing vocabulary, RT-2 overwrites the 256 least-frequently-used text tokens and reuses them as the action bins, so no new output head is added at all; the model simply learns to emit an "action sentence" when the prompt is a robot observation plus an instruction, and a normal answer otherwise. Keeping the web data in the training mixture is essential: fine-tuning on robot data alone would let the model forget its internet knowledge (catastrophic forgetting), and it is precisely that knowledge we are trying to keep.

The result is emergent semantic generalization the RT-1 data could never supply. RT-2 can act on objects, categories, and relations that appear nowhere in its robot demonstrations: "pick up the extinct animal" (the toy dinosaur), "move the banana to the sum of two plus one" (it reads the numbers), "move the Coke can to Taylor Swift" (it recognizes the face on a photo). None of these were demonstrated. They transfer because the visual and linguistic concepts live in the web-pretrained weights, and the shared token interface lets a manipulation decision query them. RT-2 roughly doubled RT-1's success rate on novel objects, backgrounds, and instructions in the paper's evaluations, and it can even do short chain-of-thought reasoning about which object to grasp before emitting the action tokens.

What actually transfers, and what it costs

Be precise about which knowledge crosses the gap. Web-scale training transfers semantics and grounding: recognizing objects, parsing instructions, resolving references, associating words with pixels. It does not transfer dynamics or contact skill. The internet contains almost no examples of the forces, slip, and fine motor corrections a fingertip needs, which is why the contact modalities of Chapter 57 and real robot data remain irreplaceable for the low-level control loop. RT-2 gets much smarter about what to do and only modestly better at how to move. The costs are real too: a multi-billion-parameter VLM cannot run onboard, so RT-2 was served from a cloud TPU at roughly 1 to 3 Hz, a latency and connectivity burden that the smaller open policies of the next section were designed to escape. And every claim of generalization has to be tested on genuinely held-out objects and instructions, not paraphrases of the training set, or the leakage-safe evaluation discipline from Chapter 5 is quietly violated and the "emergent" skill is really memorization.

from lerobot.common.datasets.lerobot_dataset import LeRobotDataset

# Downloads, decodes video, and returns normalized (image, instruction, action)
# samples ready for a VLA, with dataset action stats handled for you.
ds = LeRobotDataset("lerobot/berkeley_autolab_ur5")
sample = ds[0]
print(sample["observation.images.image"].shape, sample["action"].shape)

What's Next

In Section 58.3, we turn from Google's closed, cloud-served RT models to the open generalist policies (Octo and OpenVLA) that put a downloadable VLA on a single GPU. They keep RT-2's core insight, a pretrained backbone plus tokenized actions, while shrinking the model, opening the weights, and training on the pooled multi-robot data that makes one policy drive many different arms.