"A 92 percent success rate sounds like a grade. It is really a coin I flipped 25 times on one tabletop, on one Tuesday, with one lighting rig. Ask me again in a different kitchen."
A Rigorously Evaluated AI Agent
Prerequisites
This section closes Chapter 58, so it assumes the VLA policy of Section 58.1 and the model families of Sections 58.2 through 58.5, plus the sim-to-real and Open X-Embodiment data pipeline of Section 58.6. It leans on calibrated uncertainty from Chapter 18 and the leakage-safe benchmarking discipline of Chapter 65. No new probability is introduced beyond a binomial confidence interval, developed here.
The Big Picture
A VLA that scores well on a demo is not a VLA you can deploy. Three questions decide whether a learned policy earns a place next to a person or a payload: does the reported number mean anything (evaluation), does it hold when the world moves off the training distribution (generalization), and what happens on the trials it fails (safety). These are not afterthoughts bolted onto a finished model; they are the part of robot learning where most of the honest difficulty lives, because a policy that maps pixels to motor torques can hurt someone, and because success rate is a noisy statistic collected on physical hardware that no two labs configure the same way. This section gives you the measurement tools (confidence intervals on tiny trial counts, a taxonomy of distribution shift, sim-and-real benchmark protocols) and the runtime tools (action constraints, uncertainty-gated fallback) that turn a promising checkpoint into a system you can stand next to.
Evaluating a policy that acts
Classification is scored on a held-out set you can store in a file. A VLA is scored by running it, many times, on hardware or in a simulator, and counting task successes. That single fact drives everything hard about VLA evaluation. First, the sample size is tiny: a real-robot evaluation of 25 or 50 rollouts per task is common, because each rollout costs a minute of wall-clock and a human to reset the scene. A reported "88 percent" from 25 trials is 22 successes, and the 95 percent Wilson confidence interval on that runs roughly from 70 to 96 percent. Comparing two policies at 88 versus 80 percent on 25 trials each is comparing noise. The discipline of Chapter 65 applies with force here: report intervals, fix the number of trials before you look, and never tune on the eval scenes.
Second, the protocol is the experiment. A success is whatever a rubric says it is (object in bin, drawer fully closed), and small choices (start-pose randomization, distractor objects, the exact instruction wording) swing the number more than the model does. This is why real-robot results are famously hard to reproduce across labs, and why simulation benchmarks matter: LIBERO and CALVIN fix the scenes, resets, and success detector so two policies face the identical gauntlet, and the harness we build in Lab 58 runs against exactly these. Simulation buys reproducibility and thousands of cheap trials; it costs realism, because a policy can exploit rendering artifacts a real camera never produces. The bridge is a real-to-sim evaluator like SIMPLER, which tunes a simulator so that measured sim success rank-correlates with real success, letting you screen checkpoints cheaply before spending robot time.
Key Insight
The headline metric of a VLA is a binomial proportion estimated from a small n on a specific scene distribution. Two consequences follow that beginners routinely miss. (1) Precision scales as \(\sqrt{p(1-p)/n}\), so halving the interval needs four times the rollouts; there is no cheap way to make a 25-trial number trustworthy. (2) The scene distribution is part of the metric, not a footnote. "92 percent success" is meaningless without the objects, lighting, and instruction set it was measured on, exactly because generalization (the next subsection) is the property you actually care about, and a single in-distribution number tells you nothing about it.
Measuring generalization: the axes of shift
"Does it generalize?" is too coarse to test. VLA evaluation decomposes generalization into named axes of distribution shift, each held out deliberately so a single number becomes a profile. The standard axes: visual (new backgrounds, lighting, camera pose, distractors), object (unseen instances or categories of the same task), instruction (paraphrases and novel phrasings of a known skill), task (novel compositions or long-horizon chains of known skills), and embodiment (a different robot arm or gripper than any in training). A policy can be strong on visual shift and collapse on object shift; reporting one aggregate hides the failure you will hit in the field. This axis-by-axis view is the manipulation-domain instance of the general theory in Chapter 66, and it is where the web-scale pretraining of RT-2 and OpenVLA pays off: semantic priors from the vision-language backbone transfer to unseen objects and instructions far better than to unseen embodiments, whose motor statistics no web image teaches.
Practical Example: a warehouse pick-and-place fleet
An e-commerce operator fine-tunes an open VLA to pick items from a tote into a shipping box across 40 stations. Lab acceptance shows 94 percent on 50 trials, and the fleet ships. Week one, station 12 (near a skylight) drops to 71 percent by mid-afternoon: a pure visual shift from moving sunlight the acceptance suite never sampled. Week two, a new SKU (a mirror-finish cosmetics tin) fails 3 of 4 grasps: object shift onto a specular surface outside training. The fix was not a better model but a better evaluation: the team rebuilt acceptance as a per-axis matrix (lighting sweep, held-out SKU set, paraphrased pick instructions), each with a fixed trial count and a Wilson interval, and gated deployment on the worst axis rather than the average. The average had been hiding both cliffs.
Safety: wrapping a stochastic policy in guarantees
A VLA is a probabilistic function whose output is torque. You cannot make behavior cloning provably correct, so safety is enforced around the policy, at the interface where its action becomes motion, by a layer that is simple enough to trust. Three mechanisms compose. Action-space constraints clamp every command to a safe envelope: per-step velocity and displacement limits, a workspace bounding box, force and torque ceilings enforced by the low-level controller. This turns "the model must never command a dangerous motion" (unverifiable) into "no command leaves the envelope" (a two-line check). Runtime monitors watch execution and trip a stop on contact-force spikes, joint-limit approach, or a watchdog timeout, the embodied cousin of the functional-safety guards in Chapter 68. Uncertainty gating uses the policy's own confidence, calibrated per Chapter 18: when the action distribution is diffuse or an ensemble disagrees, the robot slows, holds, or asks a human rather than committing to a guess. A policy that knows when it does not know is worth more in a shared space than one that is slightly more accurate on average.
import numpy as np
from scipy.stats import norm
def wilson_interval(successes, n, conf=0.95):
"""95% CI for a success RATE from a small number of rollouts."""
if n == 0:
return (0.0, 1.0)
z = norm.ppf(1 - (1 - conf) / 2)
p = successes / n
denom = 1 + z**2 / n
center = (p + z**2 / (2*n)) / denom
half = z * np.sqrt(p*(1-p)/n + z**2/(4*n**2)) / denom
return (max(0.0, center - half), min(1.0, center + half))
def safe_action(mean_action, ensemble_std, env_low, env_high,
unc_threshold=0.03):
"""Clamp to a safe envelope; gate to a HOLD when the policy is unsure."""
action = np.clip(mean_action, env_low, env_high) # hard envelope
if float(np.max(ensemble_std)) > unc_threshold: # uncertainty gate
return np.zeros_like(action), "HOLD_ASK_HUMAN"
return action, "EXECUTE"
lo, hi = wilson_interval(22, 25) # 22/25 successes
print(f"88% point estimate, 95% CI = [{lo:.2f}, {hi:.2f}]")
env_low = np.array([-0.05,-0.05,-0.05, 0.0])
env_high = np.array([ 0.05, 0.05, 0.05, 1.0])
act, mode = safe_action(np.array([0.09,-0.01,0.0,1.0]), # dx exceeds envelope
np.array([0.01,0.01,0.05,0.0]), # dz very uncertain
env_low, env_high)
print(mode, act)The code above pairs the two ideas of this section. The wilson_interval call turns "88 percent" into an honest range you can compare against a threshold; the safe_action wrapper is the deployable safety envelope, clamping the out-of-range command and returning a HOLD_ASK_HUMAN mode when the per-dimension ensemble spread crosses a calibrated bound.
Research Frontier
As of 2026 the field is converging on scalable, standardized VLA evaluation. SIMPLER established real-to-sim rank correlation as a screening tool; community efforts now push toward shared, hosted evaluation so that reported numbers are comparable across labs rather than each lab's private tabletop. On the safety side, uncertainty-aligned planning (the KnowNo line of work) uses conformal prediction to give a VLA a statistically calibrated "ask for help" trigger with a bounded task-failure rate, moving safety from a hand-tuned threshold toward a guarantee. The open problem remains embodiment generalization: no current benchmark cleanly predicts how a policy trained on one arm will perform on another, and cross-embodiment transfer (Open X-Embodiment, GR00T) is measured but not yet reliably forecast.
Right Tool: don't hand-roll the eval harness
The rollout loop, per-axis scene generation, success detector, and metrics aggregation are the same across projects and easy to get subtly wrong (a mismatched success rubric silently inflates every number). The LIBERO and CALVIN benchmark suites, the SIMPLER real-to-sim evaluator, and the LeRobot evaluation utilities ship all of it: standardized task sets, deterministic resets, a scored success detector, and batched parallel rollouts. What is roughly 150 to 250 lines of environment wiring, seeding, and metric bookkeeping collapses to loading a benchmark, calling evaluate(policy, suite), and reading back per-task success with confidence intervals. Write the loop once to understand it (the Wilson interval above is the piece worth owning), then let the suite own reproducibility so your numbers are comparable to published ones.
Exercise
(1) Using wilson_interval, find the smallest number of rollouts \(n\) at which an observed 90 percent success rate has a 95 percent lower bound above 80 percent. Explain to a skeptical manager why the common "run 20 trials" budget cannot support that claim. (2) Extend safe_action into a three-tier gate: EXECUTE, SLOW (scale the action by 0.3), and HOLD, driven by two uncertainty thresholds. Then simulate 200 actions with uncertainties drawn from a distribution that is higher near a workspace boundary, and report what fraction of near-boundary actions are caught by the SLOW or HOLD tiers versus interior actions. (3) Design a per-axis generalization matrix for the warehouse example: name the held-out set for each of the five shift axes and the trial count you would fix per cell.
Self-Check
- Why is a raw success rate from 25 real-robot rollouts a weak basis for claiming one VLA beats another, and what should be reported instead?
- Name the five axes of generalization shift and give a concrete field failure that a single aggregate success rate would hide.
- Safety wraps the policy rather than fixing it. State the three enforcement mechanisms and explain why each is placed outside the learned network instead of trained into it.
Lab 58
Fine-tune or evaluate an open VLA policy on a manipulation benchmark subset (LIBERO/CALVIN). Load a pretrained OpenVLA or Octo checkpoint, run a fixed set of rollouts on a benchmark task suite, and report per-task success with Wilson confidence intervals. Then hold out one generalization axis (a distractor-heavy visual variant or a paraphrased instruction set) and quantify the drop. Finally, wrap the policy in the safe_action envelope-and-gate from this section and measure how the uncertainty gate trades success rate against unsafe-command rate.
Bibliography
Benchmarks and evaluation
The standard fixed-scene manipulation benchmark used throughout this section; its deterministic resets and success detector are what make cross-policy comparison meaningful.
Introduces long-horizon, language-conditioned evaluation with instruction and task compositional shift, the source of the "task" generalization axis.
Builds simulators whose success rates rank-correlate with real hardware, the real-to-sim screening tool that lets you triage checkpoints before spending robot time.
Policies and generalization
Kim, Pertsch, Karamcheti, et al. (2024). OpenVLA: An Open-Source Vision-Language-Action Model.
Reports a careful per-axis generalization study on an open checkpoint, the model most readily fine-tuned and evaluated in Lab 58.
Octo Model Team (2024). Octo: An Open-Source Generalist Robot Policy. RSS.
A generalist policy whose evaluation spans multiple embodiments, illustrating why embodiment shift is the hardest axis to forecast.
The cross-embodiment dataset and evaluation that defines the embodiment-generalization problem this section flags as still open.
Safety and calibrated uncertainty
Uses conformal prediction to give an embodied policy a statistically calibrated "ask a human" trigger with a bounded failure rate, the principled version of the uncertainty gate.
The reference for distribution-free intervals underlying the calibrated gating threshold; pairs directly with Chapter 18.
What's Next
Evaluation and safety told you whether a VLA is trustworthy; the next question is whether it can run where it must. In Chapter 59, we open Part XII and turn to edge AI fundamentals and model optimization: quantization, pruning, and compilation that shrink a multi-billion-parameter policy to fit the latency and power budget of a robot's onboard computer, without discarding the safety envelope this section built.