Flow-matching and open-world VLAs (π0/π0.5)

"They asked for a policy that could fold laundry. I asked which laundry. They said all of it. That was the first honest spec I had ever been handed."

An Unfolding AI Agent

Why flow matching for actions

Manipulation demonstrations are multimodal and high-dimensional. There are several equally good ways to grasp a mug, a mean-squared-error head averages them into a motion that grasps nothing, and the field's fix (Section 58.3) was to sample from a generative head. Diffusion works but pays a tax: sampling walks backward through many noise levels, so a policy that must issue actions at 50 Hz cannot afford twenty denoising steps per chunk. Flow matching is the answer π₀ reaches for. Instead of learning to reverse a stochastic noising process, it learns a deterministic velocity field that transports a sample of pure noise straight to a data sample along a nearly linear path, so a few large integration steps suffice.

Concretely, take a clean action chunk \(\mathbf{a}_1\) (a short horizon of future robot commands) and a noise sample \(\mathbf{a}_0 \sim \mathcal{N}(0, I)\). Define the interpolant \(\mathbf{a}_\tau = (1-\tau)\,\mathbf{a}_0 + \tau\,\mathbf{a}_1\) for flow time \(\tau \in [0,1]\). The target velocity along this straight path is simply the displacement \(\mathbf{a}_1 - \mathbf{a}_0\), and the network \(v_\theta\), conditioned on the observation \(\mathbf{o}\) and instruction \(\ell\), is trained to regress it:

At inference you draw noise and integrate the learned field forward from \(\tau=0\) to \(\tau=1\) with a few Euler steps. That is the whole trick: a plain regression loss (no score matching, no noise schedule to tune) that yields a fast continuous sampler. Because the loss is an expectation over flow time \(\tau\), one network learns to denoise at every level at once.

π₀: a VLM trunk with a flow-matching action expert

π₀'s architecture is two coupled transformers. The trunk is a pretrained vision-language model (PaliGemma, a 3B-parameter VLM) that ingests one or more camera images and the language instruction and provides the semantic grounding, the web knowledge that lets "the striped towel" mean something. Bolted alongside it is a smaller action expert (roughly 300M parameters) that receives the robot's proprioceptive state and the noisy action chunk, attends into the VLM's features, and outputs the flow-matching velocity. The two share attention but keep separate weights, so the language backbone is not disturbed by the fast-changing, continuous action stream. The policy predicts a chunk of \(H\) future steps (about 50 actions, half a second at 50 Hz) conditioned on recent observations:

Three properties make it a foundation model rather than a single-robot controller. It is cross-embodiment: trained on a large mixture spanning single-arm, dual-arm, and mobile manipulators, with the action-vector dimensionality padded to a common width so one head serves many bodies, exactly the flexibility argued for in Chapter 57. It is high-frequency: the flow head's cheap sampling sustains 50 Hz control, enough for the closed-loop feedback that contact-rich tasks demand. And it is post-trainable: a broad pretraining phase on the full mixture, followed by fine-tuning on a curated high-quality set for the target task, mirrors the pretrain-then-align pattern of language models. Leakage-safe evaluation (Chapter 65) is what keeps the reported success rates on those tasks honest, since the training mixture is large enough to accidentally contain your test scene.

π_0.5: co-training for open-world generalization

π₀ is dexterous but still generalizes best inside its training distribution. π_0.5 targets the harder prize: performing long-horizon tasks in entirely new environments, a real home the robot has never entered. The core idea is co-training on heterogeneous data. Alongside robot trajectories, the model trains on web image-text data, on high-level semantic subtask labels, on verbal instructions, and on data from other robots, so that the visual and semantic representations are forced to generalize far beyond the specific homes in the robot set. The claim, supported by the released results, is that this mixture is what transfers: a policy that has seen the breadth of the web plus the physics of many robots can enter an unseen kitchen and still parse the scene into actionable structure.

π_0.5 also runs a hierarchical (dual-process) inference. At each step the same model first predicts a high-level semantic subtask in words ("pick up the plate," "open the drawer"), then conditions the flow-matching action expert on that subtask to produce the low-level motion. This mirrors the perception-to-action decomposition of Section 58.1 but keeps both levels inside one network, so the semantic plan and the motor command stay coupled. The result is a policy demonstrated cleaning multiple unseen kitchens and bedrooms end to end, tasks whose horizon (many minutes, dozens of subgoals) would compound a low-level-only policy into failure.

The snippet below implements the flow-matching loss and sampler in isolation, so the mechanism referenced above is concrete rather than a black box. It is deliberately backbone-free: swap the toy MLP for a VLM-conditioned expert and this is the π₀ action head.

import torch, torch.nn as nn

class FlowActionHead(nn.Module):
    def __init__(self, action_dim, cond_dim, hidden=256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(action_dim + cond_dim + 1, hidden), nn.SiLU(),
            nn.Linear(hidden, hidden), nn.SiLU(),
            nn.Linear(hidden, action_dim))          # predicts velocity

    def velocity(self, a_tau, tau, cond):
        x = torch.cat([a_tau, tau, cond], dim=-1)
        return self.net(x)

def flow_loss(head, a1, cond):                       # a1: clean action chunk
    a0  = torch.randn_like(a1)                        # noise sample
    tau = torch.rand(a1.size(0), 1)                   # flow time in [0,1]
    a_tau  = (1 - tau) * a0 + tau * a1                # straight interpolant
    target = a1 - a0                                  # constant path velocity
    return ((head.velocity(a_tau, tau, cond) - target) ** 2).mean()

@torch.no_grad()
def sample(head, cond, action_dim, steps=8):         # few-step Euler integration
    a = torch.randn(cond.size(0), action_dim)
    for k in range(steps):
        tau = torch.full((cond.size(0), 1), k / steps)
        a = a + (1.0 / steps) * head.velocity(a, tau, cond)
    return a                                          # -> continuous action chunk

Two cautions carry over from open policies and matter more here. First, observation-distribution match: open-world generalization is real but bounded by what the co-training mixture covered, and a camera viewpoint or lighting far from the training set still degrades gracefully into failure, the distribution-shift theme of Chapter 66. Second, sampler step count: too few Euler steps and the chunk is under-integrated and jerky, too many and you lose the latency advantage that justified flow matching in the first place; five to ten is the usual sweet spot and is worth tuning per robot.

What's Next

In Section 58.5, we scale the trunk and the body together: humanoid and general-robot foundation models like GR00T and Gemini Robotics take the flow-matching and co-training ideas here and ask what changes when the embodiment is a full humanoid with many more degrees of freedom and the backbone is a frontier multimodal reasoner.