Humanoid and general robot foundation models (GR00T, Gemini Robotics)

"You gave it a body with forty joints and asked it to think fast and move faster. So it grew two brains and stopped arguing with itself."

A Two-Minded AI Agent

Why a humanoid needs a different kind of model

A generalist tabletop policy assumes a fixed camera, a 7-DoF arm, and a stable base. A humanoid breaks all three assumptions at once. The what changes: the action space is now a high-dimensional whole-body command (both arms, hands with many fingers, torso, neck, and a base or legs), and the observations arrive from cameras that move because the robot moves its own head. The why it is hard is data scarcity multiplied by dimensionality. Collecting demonstrations for a 40-DoF system by teleoperation is far slower than for a single arm, so the naive path of "record a million trajectories on this exact robot" is closed. And the balance coupling means a policy cannot treat the arms in isolation; a reach that shifts the center of mass must be compensated elsewhere in the same command. The design response is to stop asking one network to do everything at once and instead separate the timescales of cognition and control, an idea borrowed almost directly from the fast-and-slow account of human thinking.

GR00T: two systems and a data pyramid

NVIDIA's Isaac GR00T N1 is an open foundation model for humanoids built around an explicit System 2 / System 1 split. System 2 is a vision-language model (an Eagle-family VLM) that runs slowly, perhaps a few hertz, and produces a semantic latent: it reads the cameras and the instruction and decides what to do. System 1 is a diffusion transformer action head that runs fast, well above 100 Hz, conditioned on that latent plus the robot's proprioception, and generates the smooth motor chunks that actually move the joints. The two are trained together but decoupled in the loop, so the expensive VLM does not gate the control rate. The action head factorizes as a chunk distribution over a horizon \(H\), conditioned on the slow latent \(\mathbf{z}\), the recent observations, and the proprioceptive state \(\mathbf{q}\):

The second idea is the data pyramid. The base is the widest and cheapest: internet-scale human video, from which the model learns how hands approach objects and how tasks decompose, with latent actions inferred rather than measured. The middle is synthetic data generated in simulation and by neural trajectory augmentation, the digital-twin and domain-randomization machinery of Chapter 55. The narrow, precious top is real teleoperated robot data. Pretraining flows bottom to top so the scarce real data mostly has to align an already-competent model to a specific body rather than teach manipulation from nothing. Because it is openly released, GR00T is also the natural on-ramp for a lab that wants to study the recipe rather than admire it from outside.

Gemini Robotics: embodied reasoning meets dexterous control

Google DeepMind's Gemini Robotics takes a large multimodal model, Gemini, and pushes it into the physical world along two tracks. Gemini Robotics-ER (embodied reasoning) is a vision-language model tuned to reason about space, contacts, and affordances: it can point to where a grasp should go, estimate object poses, and predict trajectories, without directly emitting motor commands. Gemini Robotics proper is the vision-language-action model that closes the loop and produces actions, and it is notable for dexterity (folding, wrapping, fine bimanual manipulation) and for steerability, meaning it follows nuanced natural-language corrections mid-task. Two properties matter most for a sensing-systems reader. First, strong generalization: because the backbone inherits Gemini's world knowledge, the policy handles objects, instructions, and environments outside its robot training set, the open-vocabulary transfer argued for in Chapter 21. Second, an explicit on-device variant, Gemini Robotics On-Device, runs the policy locally on the robot for low latency and offline operation, which pulls the whole architecture squarely into the edge-deployment constraints of Chapter 59.

The snippet below shows that inference path for GR00T: a multi-camera-plus-proprioception observation and a language goal in, a whole-body action chunk out, ready for the low-level controller. It is the code behind the assembly-cell story above.

import numpy as np
from gr00t.model.policy import Gr00tPolicy
from gr00t.data.embodiment_tags import EmbodimentTag

# Load the open humanoid foundation model with its embodiment-specific transforms.
policy = Gr00tPolicy(
    model_path="nvidia/GR00T-N1.5-3B",
    embodiment_tag=EmbodimentTag.GR1,          # selects joints + action stats
    device="cuda",
)

# One observation step: head + wrist cameras, joint state, and the instruction.
obs = {
    "video.ego_view":   np.zeros((1, 256, 256, 3), dtype=np.uint8),   # head cam
    "video.wrist_view": np.zeros((1, 256, 256, 3), dtype=np.uint8),   # wrist cam
    "state.joints":     np.zeros((1, 40), dtype=np.float32),          # proprioception
    "annotation.human.task_description": ["seat the clip on the left tab"],
}

action = policy.get_action(obs)          # runs System 2 (slow) + System 1 (fast)
print(action["action.joints"].shape)     # -> (H, 40): a whole-body action chunk

Two failure modes deserve naming, both amplified by the extra degrees of freedom. First, embodiment-tag mismatch: point the policy at the wrong joint layout or action statistics and it emits confident, physically wrong whole-body commands, with no signal in the loss to warn you, the higher-dimensional cousin of the action-space mismatch from Section 58.3. Second, proprioception drift: humanoid policies lean heavily on joint-state inputs for balance and coordination, so an uncalibrated encoder or a stale state estimate corrupts the fast head even when the cameras look fine, which is exactly why the proprioceptive-sensing discipline of Chapter 57 is load-bearing here and not a detail.

What's Next

In Section 58.6, we open up the data that feeds every model in this chapter: sim-to-real transfer and the Open X-Embodiment collection, where dozens of labs pooled robot trajectories into one dataset and where the domain gap between a simulator and a real sensing rig becomes the thing you have to engineer around.