Validation of synthetic-from-real pipelines

"A synthetic frame that looks perfect to me and teaches my detector nothing is not data. It is wallpaper."

A Skeptical AI Agent

This is an advanced, research-frontier section. It assumes the splatting and re-simulation pipeline of Sections 51.2 through 51.5. It leans hard on two running threads of the book: leakage-safe evaluation, treated in general in Chapter 5 and formalized as benchmark protocol in Chapter 65, and distribution shift, treated in Chapter 66. If those chapters are unfamiliar, read their opening sections first; the whole point here is that a synthetic-data pipeline is a new and sneaky way to leak the test set into training.

Three layers of validation, and why you need all three

Validation of a synthetic-from-real pipeline stacks in three layers, from cheap and weak to expensive and decisive. The fidelity layer asks, per sample, how close a re-simulated reading is to a real one at the same pose: this is image and geometry error, and it is what neural-rendering papers report. The distributional layer asks whether the set of synthetic samples covers the real sensor distribution, because a pipeline can render every individual frame beautifully and still only ever produce sunny noon scenes. The task layer asks the one question that actually matters for a data engine: if you train a perception model on the synthetic data and test it on real held-out data, does it get better. Each layer catches failures the layer below is blind to. A pipeline can pass fidelity and fail distribution; it can pass both and still fail the task, because the errors it does make happen to sit exactly where your detector is sensitive. Never report only the first layer and call the pipeline validated.

Fidelity: measuring the sim-to-real gap per sample

Fidelity metrics come in two families matched to the two sensor modalities this chapter re-simulates. For camera output you hold out a real image at a known pose, render the scene from that exact pose, and compare. Pixel-space peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) measure low-level agreement, but they reward blur and miss the perceptual errors that break a detector, so pair them with a learned perceptual metric such as LPIPS, which correlates far better with how a downstream network sees the image. For lidar you compare the re-simulated point cloud against the real return: geometric error as a symmetric Chamfer distance or per-ray range error, plus intensity error if your model predicts reflectance, and, critically, a drop or spurious-point rate, because a re-simulator that silently deletes returns on dark or specular surfaces will teach a segmenter that those surfaces do not exist. The physics of why lidar drops returns on retroreflectors and wet asphalt is exactly the measurement-model material of Chapter 2, and a good re-simulator must reproduce those dropouts, not paper over them.

The subtlety that trips people up is that these are novel-view metrics, and they only mean something on poses genuinely held out of fitting. A splat evaluated on one of its own training views reports a gorgeous PSNR that certifies nothing. So the fitting split must reserve a set of frames the optimizer never touched, and fidelity is reported only there. The helper below computes the standard camera trio the honest way and enforces the held-out contract by refusing to score a pose that appears in the training set.

import numpy as np

def novel_view_fidelity(render, real, pose_id, train_pose_ids):
    """Per-sample camera fidelity, but only on genuinely held-out poses."""
    if pose_id in train_pose_ids:
        raise ValueError(f"pose {pose_id} was used to fit the scene: leaked metric")
    mse = np.mean((render.astype(np.float64) - real) ** 2)
    psnr = 10.0 * np.log10((255.0 ** 2) / max(mse, 1e-9))
    # SSIM / LPIPS omitted for brevity; use skimage.metrics + lpips in practice
    return {"psnr_db": round(psnr, 2), "mse": round(mse, 2)}

held_out = {17, 42, 88}          # frames reserved from fitting
train_ids = set(range(200)) - held_out
print(novel_view_fidelity(render_frame_42, real_frame_42, 42, train_ids))
# -> {'psnr_db': 24.7, 'mse': 220.1}

Distribution and coverage: does the synthetic set span the real one

Per-sample fidelity is necessary but blind to the failure that most often ruins a data engine: mode collapse in what you choose to render. A pipeline that faithfully re-simulates only the conditions present in your source logs will produce a mountain of synthetic frames that are individually perfect and collectively redundant. Distributional validation treats the real and synthetic sets as two samples from two distributions and asks how far apart they are. In feature space, the Frechet Inception Distance (FID, and its cleaner successor for reused features) compares the mean and covariance of a perception backbone's activations on real versus synthetic images; for lidar, compare occupancy or point-density histograms and the distribution of returns per range bin. Beyond a single distance you want the two directional numbers of precision (are synthetic samples realistic, that is, do they fall inside the real manifold) and recall (does the synthetic set cover the real manifold, or has it collapsed onto a mode). A rare-event generator (Section 51.5) is precisely a bet on recall: you are manufacturing regions of the distribution that the real logs undersample, so you must measure that you actually reached them and did not just re-render the easy middle.

The task layer, leakage, and calibration

The task layer runs the TSTR and TRTS experiments and reads them against a real-only baseline. The number to move is downstream performance on the held-out real test set, whatever your task metric is: mean average precision for a detector, intersection-over-union for a segmenter. A synthetic pipeline earns its keep when real-plus-synthetic beats real-alone on that metric, especially on the tail slices you built the synthetic data to cover. Two guardrails make this trustworthy. First, leakage discipline: the drives whose logs were fitted into scenes must be entirely disjoint from the drives in the real test set, split by session and geography, or you are testing whether a splat can memorize, which it can. Second, calibration: a model trained partly on synthetic data often becomes overconfident on the synthetic-looking part of the real distribution, so re-check its calibration with the reliability diagrams and conformal methods of Chapter 18. A pipeline that raises accuracy while wrecking calibration has not passed; it has moved the problem somewhere a naive accuracy number will not show it.

What's Next

In Chapter 52, the same reconstructed scenes stop being an offline data factory and start being built online: SLAM and Spatial AI turn the fit budget into a live mapping loop, and the validation instincts you built here (held-out poses, geometric error, no frame-level leakage) become the trajectory and map-quality benchmarks that decide whether a robot actually knows where it is.