"In simulation my policy has a 99 percent success rate. In the lab it has a 99 percent chance of teaching me something about friction."
A Recently Humbled AI Agent
Prerequisites
This section builds on the generalist-policy models of Section 58.3 and Section 58.4, which consume the data pooled here. It leans on the synthetic-data and digital-twin methods of Chapter 55, the leakage-safe dataset discipline of Chapter 5, and the distribution-shift vocabulary of Chapter 66. The robot sensing rig whose viewpoints must survive the transfer is the subject of Chapter 57.
The Big Picture
Every VLA in this chapter has the same appetite: it wants millions of diverse, action-labeled trajectories, and the physical world hands them over one slow teleoperated episode at a time. There are only two ways to feed that appetite at scale. You can pool real data across many labs and robots, which is what Open X-Embodiment does, turning 22 incompatible robot fleets into one training corpus. Or you can generate data in simulation, which is nearly free but systematically wrong in ways that break policies the instant they touch hardware. This section is about both levers and the seam between them: what the Open X-Embodiment collection actually contains and how to consume it, why the sim-to-real gap exists, and the domain-randomization and system-identification tricks that let a policy trained on a fiction survive reality.
Two ways to feed a data-hungry policy
A manipulation policy needs to see the long tail of the physical world: odd object shapes, unfamiliar lighting, cluttered scenes, and the thousand small perturbations that a curated lab never produces. Real robot data captures that tail faithfully but at brutal cost, because every trajectory requires a physical robot, a human teleoperator, and wall-clock time that does not compress. Simulation inverts the economics. A GPU farm can roll out millions of episodes overnight, with perfect state labels, automatic domain variation, and no risk of a dropped mug. The catch is the reality gap: the simulator's contact model, friction, sensor noise, and rendering never quite match the physics your robot lives in, so a policy that memorizes the simulator's quirks fails on the real one. The design problem of this section is choosing where each trajectory should come from and how to keep simulated data honest enough to transfer.
Open X-Embodiment: pooling the world's robot data
Open X-Embodiment (OXE), released in 2023 by a consortium of 21 institutions, is the ImageNet moment for manipulation. It aggregates 60 existing robot datasets into one standardized collection: roughly one million real trajectories spanning 22 distinct robot embodiments, from single-arm grippers to bimanual systems and mobile manipulators, covering more than 500 skills and 150,000 tasks. The point is not the raw size alone; it is the demonstration that cross-embodiment training works. The accompanying RT-X models (RT-1-X and RT-2-X) showed that a single policy trained on the pool outperforms specialist policies trained on each robot's own data in isolation, with reported success improvements around 50 percent on several platforms. Skills learned on one arm transfer to another, which means the marginal value of your 300 demonstrations is far higher when they join the pool than when they train a model alone.
Consuming OXE forces you to confront why pooling is hard. Each source dataset has its own camera count and placement, its own control frequency, its own action convention (joint velocities here, end-effector deltas there), and its own gripper semantics. OXE standardizes the container by shipping everything as RLDS (Reinforcement Learning Datasets, a TFDS-based episodic format), but the content still needs normalizing before a model can learn from it. Actions are typically converted to a common 7-DoF end-effector frame and normalized per dataset (often to the 1st and 99th percentile range so outliers do not dominate the scale), and observations are resampled to a shared resolution and frame rate. This is the same leakage-safe curation logic argued in Chapter 5, applied to a corpus assembled from strangers' hard drives.
Key Insight
Cross-embodiment transfer works because manipulation shares structure that survives changes of body. The semantics of "pick up the red block" and the geometry of reaching, grasping, and lifting are largely embodiment-independent; only the final map from intent to joint commands is robot-specific. Pooling lets the model learn the shared 95 percent from everyone's data and reserve fine-tuning for the robot-specific 5 percent. This is exactly why an open checkpoint plus a few hundred local demonstrations beats training from scratch: the expensive part was already paid for, once, by the whole community.
Crossing the reality gap: randomize what you cannot measure
When real data runs out, simulation fills the gap, and two complementary techniques keep the fiction transferable. The first is domain randomization: instead of trying to build one perfectly accurate simulator, you deliberately vary its parameters (textures, lighting, camera pose, object mass, friction coefficients, and actuator gains) across a wide range during training. If the policy performs across that whole range, the real world becomes just one more sample from a distribution it already handles, and the reality gap looks like ordinary variation rather than a cliff. Formally, you train to minimize expected loss over a distribution \(p(\xi)\) of simulator configurations,
$$\theta^\star = \arg\min_\theta \; \mathbb{E}_{\xi \sim p(\xi)}\big[\, \mathcal{L}(\pi_\theta; \xi) \,\big],$$so that the real environment \(\xi_{\text{real}}\) sits inside the support of \(p(\xi)\). The second technique is system identification: measure the real robot's dynamics and sensor noise, then tune the simulator so its nominal parameters match, shrinking the gap you then randomize around. Randomization without identification wastes capacity on physically impossible worlds; identification without randomization overfits to one operating point. Used together they bracket reality, and the digital-twin construction of Chapter 55 is precisely how you build the identified simulator to randomize around.
In Practice: a robotics team ships a bin-picking cell without breaking hardware
A logistics-automation team needs a policy that empties bins of mixed parts into a sorter. Real teleoperation gives them 2,000 episodes, enough to fine-tune but not enough to cover every part orientation and every glare condition on the warehouse's reflective totes. They stand up an Isaac Sim scene of the exact cell, run system identification against the real arm's step responses to fix nominal friction and latency, then randomize lighting, tote reflectance, part mass, and camera pose across 400,000 simulated grasps. The policy is co-trained on the OXE pool plus their real 2,000 plus the simulated 400,000. On the physical cell it reaches a usable grasp rate on parts and lighting it never saw teleoperated, and, just as important, the sim rollouts absorbed the dangerous exploration so no real gripper crashed into a bin wall during learning. The lever that mattered was not any single dataset; it was mixing pooled real data, a handful of local real episodes, and heavily randomized simulation into one training set.
A minimal cross-embodiment data pipeline
The code below shows the core normalization step every OXE consumer performs: stream an RLDS dataset, extract the shared observation-action fields, and normalize actions to a common per-dataset scale so trajectories from different robots become comparable. It is the unglamorous glue that makes pooling possible, and it is referenced in the shortcut that follows.
import numpy as np
import tensorflow_datasets as tfds
# One OXE source in RLDS/TFDS form (each shard is an episode of steps).
ds = tfds.load("bridge_dataset", split="train", data_dir="gs://gresearch/robotics")
def episode_to_arrays(episode):
imgs, acts = [], []
for step in episode["steps"]:
imgs.append(step["observation"]["image"].numpy()) # H x W x 3
acts.append(step["action"].numpy()) # 7-DoF ee delta
return np.stack(imgs), np.stack(acts)
# Estimate robust per-dataset action statistics (1st/99th percentile, not min/max).
all_actions = np.concatenate([episode_to_arrays(ep)[1] for ep in ds.take(500)])
lo, hi = np.percentile(all_actions, 1, axis=0), np.percentile(all_actions, 99, axis=0)
def normalize(action):
# Map into [-1, 1] against the robust range, then clip the tails.
return np.clip(2.0 * (action - lo) / (hi - lo) - 1.0, -1.0, 1.0)Library Shortcut
Hand-rolling the OXE loader (RLDS parsing, per-dataset normalization statistics, resizing, frame-rate resampling, and the sampling weights that stop one huge dataset from dominating the batch) is roughly 400 to 700 lines across a dozen datasets. The rlds_dataset_builder and the Octo/OpenVLA data mixes reduce it to a config: you list dataset names with sampling weights and call the provided make_interleaved_dataset(...), and it returns a normalized, shuffled, embodiment-mixed stream ready for training. The heterogeneity handling that used to be a research project is now a YAML block.
Research Frontier
As of 2026 the data frontier has moved past pooling human teleoperation. DROID (a 76,000-trajectory dataset gathered on a standardized rig across 500-plus scenes) pushes real-world diversity, while the sim side is scaling automatic data generation: NVIDIA's Isaac GR00T pipeline and DexMimicGen synthesize large trajectory sets from a few human demonstrations, and RoboCasa and MimicGen generate task and scene variety procedurally. The active question is the real-to-sim ratio: how much cheap randomized simulation you can add before it dilutes the real signal, and whether learned simulators (the world models of Chapter 53) can close the reality gap that hand-built physics engines cannot.
Exercise
You are assembling a training mix from three OXE datasets: one with 400,000 trajectories on a single arm, one with 20,000 on a bimanual robot, and your own 500 on a target arm. (1) Explain why uniform sampling over trajectories is the wrong batching strategy and propose a weighting that preserves the small datasets' influence. (2) Your target robot uses joint-position control but two source datasets use end-effector deltas. Describe what must happen before their actions can share a normalization. (3) You add 300,000 domain-randomized simulated grasps. State one concrete leakage risk this introduces into your held-out real evaluation and how you would prevent it, referencing the split discipline of Chapter 65.
Self-Check
1. What does Open X-Embodiment standardize, and what does it deliberately leave for the consumer to normalize?
2. Why do domain randomization and system identification work better together than either alone?
3. Why is robust 1st/99th-percentile normalization preferred over min/max scaling when pooling actions across robots?
What's Next
In Section 58.7, we turn from feeding these policies to judging them: evaluation protocols (LIBERO, CALVIN, and real-robot success rates), the safety envelope an embodied model must respect, and how to measure genuine generalization rather than a benchmark score that quietly leaked into the training pool assembled here.