"A low reconstruction error told me my dreams were sharp. It never told me whether the ball would fall."
A Circumspect AI Agent
Prerequisites
This section assumes the four world-model families of this chapter: the latent recurrent models of Section 53.2, the joint-embedding predictors of Section 53.3, the generative interactive models of Section 53.4, and the latent planning of Section 53.5. Evaluation leans on the calibration and proper-scoring machinery of Chapter 18 and the leakage-safe protocol discipline of Chapter 65. If "proper scoring rule" or "temporal split" is unfamiliar, read those first.
The Big Picture
A world model that produces gorgeous, sharp predictions can still be physically illiterate: it can render a bouncing ball frame by frame and then let that ball roll straight through a wall. The central problem of this section is that the loss a world model was trained on, pixel or latent reconstruction error, is almost never the property we actually want, which is a model of how the physical world evolves and reacts to actions. Evaluation therefore has to measure three distinct things that a single training loss conflates: predictive fidelity (does the forecast match what happened), physical plausibility (does the model obey the constraints of the world even on inputs it was never trained on), and downstream utility (does planning inside the model produce good actions on the real system). Physical-reasoning benchmarks, built around the "violation of expectation" paradigm borrowed from developmental psychology, are the community's sharpest instrument for the second of those, and they expose failures that every fidelity metric happily hides.
Three axes, not one number
The what of world-model evaluation is a refusal to collapse quality into a single scalar. A world model \(p_\theta(o_{t+1:t+H} \mid o_{\le t}, a_{t:t+H})\) can be scored on three orthogonal axes. Fidelity asks how close a rollout is to the ground-truth continuation, measured with pixel metrics (PSNR, SSIM, LPIPS), distributional video metrics such as Fréchet Video Distance (FVD), or, for sensor streams, forecasting scores like MASE and the continuous ranked probability score (CRPS). Plausibility asks whether the model's preference ranks the physically possible above the impossible, independent of whether it nailed the exact trajectory. Utility asks the only question a control engineer cares about: when an agent plans against the model (Section 53.5), does the resulting policy succeed on the real environment? These can diverge sharply. A diffusion video model can win on FVD yet fail plausibility; a JEPA can be mediocre at pixel fidelity (it discards pixels by design) yet top plausibility and utility. Reporting one axis and implying the others is the most common evaluation sin in the literature.
Key Insight: fidelity is necessary but never sufficient
Low reconstruction error is compatible with zero physical understanding, because the metric averages over pixels or samples that are dominated by predictable background. A model can score well by copying the static scene and blurring the one thing that matters (the moving object) into a probability-hedging smear. Violation-of-expectation testing breaks this by construction: it presents matched pairs of videos, one possible and one physically impossible, that are pixel-wise almost identical up to the critical event, and it asks only whether the model is more surprised by the impossible one. Because the pair is matched, background predictability cancels, and the score isolates the model's grasp of occlusion, solidity, continuity, and gravity. A model that reconstructs beautifully but scores at chance on matched pairs has learned appearance, not physics.
Physical-reasoning benchmarks and violation of expectation
The how of plausibility testing is a family of synthetic and semi-synthetic benchmarks with controlled physics. IntPhys (Riochet and colleagues, 2018) pioneered the violation-of-expectation (VoE) protocol for machines: it renders quadruplets of clips probing object permanence, shape constancy, and spatiotemporal continuity, half possible and half impossible, and scores a model by whether its per-frame plausibility (typically a rollout or reconstruction error used as a surprise proxy) is higher on impossible clips. GRASP and the later intuitive-physics suites extend this to more properties (solidity, gravity, inertia, collision) and to the modern practice of probing large pretrained video models zero-shot. Physion (Bear and colleagues, 2021) moves from "is this possible" to prediction: given the first frames of a scene with towers, ramps, and dominoes, will these two objects touch? It compares models to human agreement, which is the honest ceiling. PHYRE (Bakhtin and colleagues, 2019) is action-conditioned: place one ball so that the green object touches the blue one, scored by sample efficiency, so it measures utility, not just prediction. CLEVRER (Yi and colleagues, 2020) adds causal and counterfactual questions ("what happens if we remove the cylinder"), the hardest reasoning tier. The 2025 result that made this concrete for our Part: V-JEPA 2's latent prediction error spikes on IntPhys-style impossible events without any physics supervision, the first strong evidence that a self-supervised sensor world model acquires intuitive physics as a byproduct.
import numpy as np
def voe_scores(surprise_pos, surprise_imp):
"""Violation-of-expectation metrics from matched possible/impossible pairs.
surprise_pos, surprise_imp: arrays of per-clip surprise (e.g. rollout error),
aligned so index i is a matched pair."""
diff = surprise_imp - surprise_pos # want > 0 if model "gets" physics
rel = 2 * (surprise_imp > surprise_pos).mean() - 1 # signed pairwise accuracy in [-1, 1]
# Absolute (harder): rank ALL clips, ask if impossible ones are flagged as surprising
labels = np.r_[np.zeros_like(surprise_pos), np.ones_like(surprise_imp)]
scores = np.r_[surprise_pos, surprise_imp]
order = np.argsort(-scores) # most surprising first
auroc = roc_auc(labels[order][::-1]) # relative ordering vs. labels
return {"pairwise_acc": (rel + 1) / 2, # 0.5 = chance
"mean_margin": diff.mean(),
"abs_auroc": auroc}
def roc_auc(sorted_labels): # tiny rank-based AUROC
pos = sorted_labels.sum(); neg = len(sorted_labels) - pos
ranks = np.arange(1, len(sorted_labels) + 1)
return (ranks[sorted_labels == 1].sum() - pos * (pos + 1) / 2) / (pos * neg)
pairwise_acc (the "relative" protocol) uses matched pairs so background cancels and 0.5 is chance; abs_auroc (the "absolute" protocol) drops the pairing and asks the model to flag impossible clips against all clips, which is much harder and where most models drop toward chance. Reporting only the relative number, as many papers do, flatters the model.Listing 53.7 encodes a distinction that decides whether a benchmark result means anything. The relative protocol is generous because it hands the model a matched foil; the absolute protocol, where a clip must look surprising with no partner to compare against, is the one that predicts real-world anomaly detection. A model can post a strong relative score and near-chance absolute score, and the gap between them is itself the most informative single diagnostic you can report.
Metrics for prediction: pixels, latents, and distributions
For fidelity on continuous rollouts the why of metric choice is signal structure. Pixel PSNR rewards blur, so it is a poor guide for anything with irreducible detail; perceptual and distributional metrics (LPIPS, FVD) correlate better with what humans call realism. For probabilistic sensor forecasts the right tool is a proper scoring rule, which is minimized only by the true predictive distribution and therefore cannot be gamed by hedging. CRPS generalizes absolute error to distributions, $$ \mathrm{CRPS}(F, y) = \int_{-\infty}^{\infty} \big(F(z) - \mathbf{1}\{z \ge y\}\big)^2 \, dz, $$ and rewards a forecast that is both sharp and calibrated, the property Chapter 18 makes central. Crucially, all of these must be reported as a function of horizon: a curve of error versus \(H\), not a single averaged number, because world models fail by compounding, and a model that is excellent at \(H{=}1\) and diverges at \(H{=}20\) is useless for the long-horizon planning of Section 53.5. The shape of the divergence curve, and where it crosses a task-defined tolerance, is the number that governs deployment.
Practical Example: qualifying a world model for a highway driving stack
An automotive team wants to use a learned world model to imagine the next four seconds of a merge and plan lane changes. Their first evaluation reported a single averaged FVD and a proud "state of the art." It shipped a model that, in shadow mode, occasionally hallucinated a vehicle passing through the guardrail. The rebuilt protocol scored three axes separately. Fidelity became a CRPS-versus-horizon curve on held-out logs with a strict temporal split (no future frames leaking into training, per Chapter 65). Plausibility became a VoE suite of hand-authored impossible clips (a car teleporting across a barrier, a pedestrian passing through a bus), scored with the absolute protocol so a lone impossible event had to register as surprising. Utility became closed-loop planning success and, critically, collision rate when the planner acted on the model's rollouts. The guardrail hallucination that FVD averaged away showed up immediately as a near-chance absolute-VoE score on the "solid barrier" category, and the model was sent back before it reached a safety case under Chapter 43's occupancy pipeline.
Right Tool: do not hand-roll FVD or CRPS
A correct FVD needs a specific pretrained I3D feature extractor, the right frame sampling, and a numerically stable matrix square root for the Fréchet distance; a naive reimplementation silently disagrees with published numbers and makes your results incomparable. torchmetrics ships FrechetInceptionDistance, video FVD wrappers, LPIPS, and SSIM, and properscoring (or scoringrules) gives an exact, tested CRPS in one call instead of the fifteen-line ensemble-CRPS integral it is easy to get subtly wrong. Roughly 200 lines of metric plumbing and its silent-disagreement failure mode collapse to two imports and two function calls, and your numbers become comparable to the leaderboard.
From benchmark to sensor world models
The when matters because most benchmarks in this literature are rendered video, while the world models of this book run on IMU, radar, PPG, and industrial telemetry. Two adaptations carry the physical-reasoning idea across. First, the VoE construction transfers directly: to test whether a sensor world model has learned the dynamics of a machine, synthesize matched sequences where one obeys the physics of the asset and one violates it (an impossible instantaneous temperature jump, a vibration spectrum that cannot arise from the drivetrain), and score whether the model's surprise separates them, exactly as Chapter 58's embodied agents test intuitive physics. This is the same surprise signal that Section 53.6 turns into anomaly anticipation, so the natural utility metric is lead time: how many samples before a real fault does the model's prediction error cross threshold, traded against false-alarm rate on healthy runs. Second, the digital-twin validation of Chapter 55 is a fidelity test with a physics oracle: compare the model's rollout against a mechanistic simulator on trajectories neither was fit to. And because sensor world models will meet inputs outside their training regime, plausibility must be probed under the distribution shift of Chapter 66, where fidelity metrics are least trustworthy and VoE-style structural tests earn their keep.
Research Frontier: benchmarks that resist being gamed
The frontier as of 2025 to 2026 is dominated by three problems. First, metric hacking: FVD is known to be manipulable and to reward memorization, and the community is moving toward closed-loop, utility-first suites (WorldModelBench-style and PHYRE-style action-conditioned scoring) plus human-agreement ceilings a la Physion. Second, zero-shot physical reasoning in foundation video models: V-JEPA 2 and contemporaries post above-chance IntPhys relative scores with no physics supervision, but their absolute-protocol scores and their calibration remain weak, so whether "intuitive physics emerges from scale" is a genuine capability or a matched-pair artifact is actively contested. Third, evaluation of generative interactive world models (Genie-style, Section 53.4), where there is no ground-truth future because the model invents a controllable environment, forcing metrics built on action-consistency, long-horizon stability, and controllability rather than reconstruction. For sensor-domain world models the open question is a standard leakage-safe physical-reasoning benchmark at all: the video community has IntPhys and Physion, the wearable and industrial communities have nothing comparable yet.
Exercise
Take any world model you trained in this chapter (the Lab model below is fine). (1) Build a matched VoE set: generate 200 short sequences from a simple simulator (a bouncing ball with a wall, or a synthetic machine with a physical constraint), and for each make an "impossible" twin that violates one law (the ball passes through the wall) while keeping every other frame identical up to the event. (2) Use rollout error as the surprise proxy and report both metrics from Listing 53.7; explain the gap between the relative and absolute scores. (3) Plot CRPS versus horizon \(H\) on held-out real data and mark the horizon at which error crosses a tolerance you justify from a downstream task. (4) Confirm your train/test split is temporal, not random, and state one concrete way random splitting would have inflated every number.
Self-Check
1. Name the three orthogonal axes of world-model quality and give one metric for each. 2. Why does a matched-pair VoE design let a plausibility score isolate physics from appearance, and why is the absolute protocol harder than the relative one? 3. Why must fidelity be reported as a curve over horizon rather than a single averaged number, and which downstream use makes the long-horizon end decisive?
Lab 53
train a small latent world model on a sensor sequence and use it for prediction/anomaly anticipation.
Bibliography
Physical-reasoning and violation-of-expectation benchmarks
Introduced the violation-of-expectation protocol for machines: matched possible/impossible clips scored by relative surprise. The template every later intuitive-physics suite follows.
Moves from plausibility to prediction (will these objects touch?) and, crucially, benchmarks models against human agreement as the honest ceiling.
Bakhtin, A., et al. (2019). PHYRE: A New Benchmark for Physical Reasoning. NeurIPS.
Action-conditioned physical reasoning scored by sample efficiency, so it measures planning utility rather than passive prediction.
Yi, K., et al. (2020). CLEVRER: Collision Events for Video Representation and Reasoning. ICLR.
Adds causal, predictive, and counterfactual question tiers, isolating reasoning about physics from mere trajectory fitting.
World models and their evaluation
Reports that latent prediction error spikes on IntPhys-style impossible events with no physics supervision, the strongest recent evidence that intuitive physics emerges from self-supervised video pretraining.
Hafner, D., et al. (2023). Mastering Diverse Domains through World Models (DreamerV3). arXiv.
The reference latent world model whose evaluation is utility-first: performance is judged by control success across many domains, not by reconstruction quality.
Defines Fréchet Video Distance and documents its sensitivities, essential background for why single-number fidelity metrics can mislead.
Proper scoring and forecast evaluation
The foundational treatment of CRPS and proper scoring rules that cannot be gamed by hedging, the correct basis for scoring probabilistic sensor forecasts.
Ansari, A. F., et al. (2024). Chronos: Learning the Language of Time Series. arXiv.
A modern time-series foundation model whose evaluation combines CRPS-style probabilistic scores with strict leakage-safe splits, a template for scoring sensor world models.
What's Next
In Chapter 54, we stop modeling a single stream's future and start modeling relationships between sensors, treating a deployment as a graph of nodes whose dynamics couple across space and process. The evaluation instinct built here, separate fidelity from plausibility from utility and never trust one averaged number, carries straight into scoring spatiotemporal forecasts and fault localization over sensor networks.