Synthetic data generation for rare events

"The accident you never recorded is still the one your model will be judged by."

A Long-Tailed AI Agent

Why the tail is a data problem, not a model problem

Start with the what. Empirical risk minimization averages loss over whatever distribution your data came from. If a hazard class occurs with probability \(p_{\text{rare}}\), then in a dataset of \(N\) frames it contributes on the order of \(N p_{\text{rare}}\) examples, and its gradient signal is drowned by the common case. The why is arithmetic: at \(p_{\text{rare}} = 10^{-6}\) per frame, a fleet logging a billion frames a year sees roughly a thousand instances, scattered across hundreds of visually distinct sub-types, with no guarantee any given sub-type appears at all. You cannot label your way out; the events are not in the logs. This is the same distribution-shift failure mode studied in Chapter 66, except here you have a lever to move the training distribution deliberately.

The how is to treat generation as importance sampling over a hazard space. Let \(z\) parameterize a scenario (actor positions, velocities, weather, ego trajectory). The risk you actually care about up-weights dangerous regions by a task-relevance weight \(w(z)\):

$$\mathcal{R} = \mathbb{E}_{z \sim p_{\text{world}}}\big[w(z)\,\ell(z)\big] = \mathbb{E}_{z \sim q(z)}\Big[\tfrac{p_{\text{world}}(z)}{q(z)}\,w(z)\,\ell(z)\Big].$$

You choose a proposal \(q(z)\) that oversamples the tail, generate synthetic frames from it, and correct with the ratio \(p_{\text{world}}/q\) so the model learns the rare loss without being told the tail is common. The when is any safety case where the cost of a miss dwarfs its frequency: autonomous driving, industrial safety interlocks, collision-avoidance robotics. When the operating distribution is stationary and the tail is benign, plain collection is cheaper and you should not reach for this.

Four editing operators on a reconstructed scene

Rare-event synthesis reduces to a small algebra of edits applied to the Gaussian scene of Section 51.4. Actor insertion composites an asset (itself a splatted or meshed object) at a chosen pose, with the renderer resolving occlusion against the scene depth and casting a matching shadow; this manufactures jaywalkers, debris, and stalled vehicles. Trajectory perturbation re-times or re-routes an existing dynamic actor to produce the near-miss the original drive avoided: the same truck, made to cut in. Condition transfer re-renders the fixed geometry under new lighting, fog, or rain, exploiting that the scene's structure is decoupled from its appearance. Ego re-simulation renders novel viewpoints along a counterfactual ego path, exercising the sensor rig from angles the real drive never took, which is the direct payoff of the free-viewpoint property from Section 51.3.

Because you place the actor, you know its 3D box, class, and instance mask exactly; the labels are generated, not annotated, and they are perfect by construction for the lidar and camera targets of Chapter 42. The engineering risk is physical plausibility: an inserted pedestrian floating above the road, a shadow with the wrong azimuth, or a trajectory that clips a wall teaches the model artifacts instead of hazards. Snapping actors to the reconstructed ground plane and to a drivable-area map, and validating each shadow against the scene's estimated light direction, are the guardrails that keep the tail realistic rather than merely present.

import numpy as np

def sample_cutin_scenarios(n, ego_speed=14.0, rng=np.random.default_rng(0)):
    """Propose rare 'cut-in' actor trajectories oversampling the dangerous tail,
    and return importance weights that correct back to the fleet distribution."""
    # q(z): proposal deliberately biased toward short, aggressive cut-ins
    gap = rng.uniform(3.0, 25.0, n)           # longitudinal gap at cut-in (m)
    lat_speed = rng.uniform(0.5, 3.5, n)      # lateral closing speed (m/s)
    ttc = gap / np.maximum(ego_speed - rng.uniform(-2, 4, n), 1e-3)  # time-to-collision

    # fleet-observed density p_world: cut-ins are almost always gentle and distant
    p_world = np.exp(-0.5 * ((lat_speed - 0.8) / 0.4) ** 2) * np.exp(-gap / 30.0)
    q = np.ones(n)                             # near-uniform proposal over the box
    weight = p_world / (q + 1e-9)              # importance-sampling correction

    rare = ttc < 1.5                           # the safety-critical sub-population
    return dict(gap=gap, lat_speed=lat_speed, ttc=ttc, weight=weight, rare=rare)

s = sample_cutin_scenarios(10_000)
print(f"rare (TTC<1.5s) fraction under proposal: {s['rare'].mean():.3f}")
print(f"effective rare fraction after reweighting: "
      f"{s['weight'][s['rare']].sum() / s['weight'].sum():.5f}")

As Listing 51.5 shows, generation splits cleanly into a cheap scenario sampler and an expensive renderer. You iterate on the sampler, the part that decides which rare events to make, without ever touching the splatting pipeline that turns each accepted \(z\) into labeled sensor frames.

Keeping the generator from teaching its own artifacts

The failure that quietly ruins rare-event synthesis is leakage of the generator. If the same reconstructed blocks appear in both training and evaluation, or if the model learns the tell-tale rendering signature of inserted actors (a subtly different noise floor, a too-clean silhouette) rather than the hazard itself, your metrics inflate while real-world behavior does not move. The defenses are concrete. Split by source scene, not by frame, so no reconstruction straddles train and test, extending the leakage-safe discipline of Chapter 5 from recordings to reconstructions. Mix synthetic and real at a controlled ratio and monitor whether a discriminator can separate them; if it can trivially, the model can too. And always reserve a real rare-event test set, however small, as the only ground truth that counts, because a synthetic-only evaluation measures the renderer, not the world. Section 51.7 makes this validation rigorous; the point here is that the generator's convenience is also its trap.

A second, subtler risk is coverage collapse: an over-tuned proposal \(q\) that piles thousands of near-identical worst-cases teaches the model one narrow hazard and calls the tail solved. Rarity is not enough; the synthetic tail must be diverse across the axes that matter (approach geometry, actor appearance, occluder layout, sensor conditions). Stratifying the proposal over those axes, rather than sampling one scalar severity, is what turns amplification into genuine generalization.

from drivestudio import Scene, Actor          # reconstructed splat scene + asset API
scene = Scene.load("urban_block_07")           # static background from a real drive
scene.insert(Actor("pedestrian_03"),           # library asset
             traj=cutin_traj, snap_to="ground") # occlusion + shadow handled by renderer
lidar, cam, boxes = scene.render(sensor_rig)   # labeled synthetic frames, 3D boxes free

What's Next

In Section 51.6, we confront the cost side of all this rendering: reconstructing and re-simulating a scene fast enough to matter, on the real-time and memory budgets that separate an offline data factory from a system that can reconstruct on the move.