World models for physical AI

"A generated video that looks right and a future that is right are two different achievements. I am paid for the second one, and my customers can tell the difference the instant a wheel rolls uphill."

A Physically Skeptical AI Agent

From bespoke simulators to world foundation models

The what that is new is scale and generality. A classical world model trains on the exact system it will run on, which caps it at that system's diversity and forbids zero-shot transfer. A world foundation model instead pretrains on heterogeneous corpora of video and multi-sensor logs, learns dynamics broadly true of the physical world (objects persist, fluids flow, contacts resist), and is then specialized on a little target data or conditioned on a platform's actions. The why follows the foundation-model logic: prediction is a free, dense objective (Chapter 17), so the signal is as abundant as recorded reality, and the scaling laws that rewarded scale for language appear to hold for dynamics too. The how reuses Chapter 53's three blocks, encoder, transition, predictor, grown into large cross-domain networks with explicit action conditioning so the simulator is steerable rather than a passive forecaster.

Two families compete. Generative world models predict in observation space, synthesizing the next video frames or sensor sweep and paying their loss there. Joint-embedding predictive models predict only in a learned latent and never reconstruct pixels, on the argument that one should model what is predictable and ignore noise. Meta's V-JEPA 2 (2025) leads the second camp and added an action-conditioned head that let a robot plan from the pretrained representation; the generative camp is defined by the video systems below. Scale has amplified the Chapter 53 split, not resolved it.

The physics-fidelity gap: looking right versus being right

The central frontier finding is uncomfortable. A video world model can reach photorealism and temporal smoothness while systematically violating physical law: objects wink in and out behind occluders, liquids gain volume, momentum is not conserved, a ball rolls up a slope, all because the pixels look plausible and the loss never penalized the impossibility. Studies probing generative video models against controlled physical scenarios find visual quality and physical correctness only weakly coupled, with scale improving the former far faster than the latter. For entertainment this gap is harmless. For a sensing system that will act on the imagined future it is the whole problem: a trajectory that violates conservation is not a slightly worse forecast but a confident hallucination the downstream controller cannot distinguish from truth.

The gap persists because pixel reconstruction loss rewards perceptual similarity, not causal consistency, and the physically-lawful continuations form a thin manifold inside the visually-plausible ones. Three directions attack it. Physics-informed constraints bake conservation laws, contact dynamics, or a differentiable simulator in as structure or auxiliary loss, the world-model form of the hybrid principle from Chapter 55. Latent-space prediction is the joint-embedding argument that abandoning pixels removes the incentive to render pretty-but-wrong detail. 3D and occupancy-grounded models predict in a geometric state where persistence and collision are structurally easier to enforce than in raw pixels, connecting to the occupancy forecasting of Chapter 43. None has closed the gap; measuring it rigorously is itself an open problem for Section 72.6.

Beyond pixels: world models across the whole sensor stack

Physical AI is not video AI. A robot, car, or wearable perceives through inertial units, radar returns, lidar sweeps, tactile arrays, and RF channels, and a world model earns its place only if it predicts those streams, not just a camera's. The frontier is multi-sensor, action-conditioned neural simulation: given the platform's recent multimodal history and a proposed action, forecast the next lidar point cloud, the next radar Doppler map (Chapter 44), and the next IMU trace jointly and consistently. Such a learned sensor simulator manufactures unlimited labeled, physically-plausible data for training and stress-testing perception stacks, easing the label famine of every safety-critical domain, and enables closed-loop rehearsal of a policy before real-world exposure. The hard part is cross-modal consistency: a model that predicts a pedestrian in the camera latent but not in the lidar future has learned two disconnected dreams, and reconciling them inherits every problem from Chapter 48.

Listing 72.4 makes the imagination horizon measurable. It rolls a small ensemble of stochastic dynamics forward from a shared state and treats the moment the ensemble's disagreement crosses a tolerance as the point where the world model should stop being believed, exactly the discipline the driving team relies on.

import numpy as np

def imagination_horizon(step_fns, z0, tol, H=64, seed=0):
    """Roll an ensemble of world models forward from a shared latent z0.
    Return the horizon at which ensemble disagreement (std across members)
    first exceeds tol: beyond it, the imagined future is not trustworthy."""
    rng = np.random.default_rng(seed)
    states = [z0.copy() for _ in step_fns]          # one trajectory per member
    for h in range(1, H + 1):
        states = [f(s, rng) for f, s in zip(step_fns, states)]
        spread = np.stack(states).std(axis=0).mean()  # cross-member disagreement
        if spread > tol:
            return h, spread
    return H, spread

# Ensemble of nearly-identical expansive dynamics with slightly different gains.
def make_step(gain):
    A = np.array([[1.0, 1.0], [0.0, gain]])          # gain>1 => expansive, drifts
    return lambda z, rng: A @ z + 0.01 * rng.standard_normal(2)

ensemble = [make_step(g) for g in (1.03, 1.05, 1.07, 1.09)]
h, spread = imagination_horizon(ensemble, np.array([1.0, 0.2]), tol=0.5)
print(f"trust the rollout for {h} steps (disagreement {spread:.3f} at cutoff)")

Using world models, and what still blocks them

The when is where physical AI cashes the model in. Three uses dominate. In planning, a policy searches over actions by imagining consequences in the world model and picking the best, the Dreamer loop now run on a pretrained simulator; V-JEPA 2's action-conditioned planning shows this on real robots. In sim-to-real and data generation, the model manufactures labeled sensor futures for training and stress-testing, feeding the out-of-distribution work of Chapter 66. In embodied policies, the world model is the predictive backbone under a vision-language-action controller (Chapter 58), giving the agent a runnable model of consequences rather than a reflex.

What still blocks all three is a short, stubborn list. Long-horizon consistency: rollouts drift and lose object permanence, as the horizon bound predicts. Physical grounding: the looking-right-versus-being-right gap is unmeasured on any standard benchmark. Controllability: adding faithful action-conditioning to a passive video model without wrecking fidelity is unsolved outside narrow domains. Calibrated uncertainty: most systems report no confidence over their imagined future, so a consumer cannot tell dream from forecast. Compute and evaluation: training costs are extreme, and we lack agreed metrics for whether a world model's physics is correct rather than pretty. That evaluation vacuum is a first-class open problem, and Section 72.6 treats it as one.

What's Next

In Section 72.5, we let the sensing system act on its own predictions: an agentic, self-improving loop that uses a world model to decide what to measure next, requests the samples that would most reduce its uncertainty, and updates itself online. The imagination horizon built here becomes the agent's budget for how far ahead it is allowed to plan before it must go collect real evidence.