Joint lidar-camera splatting (SplatAD)

"Build the world once. Then let each sensor look at it the way its own optics demand."

A Frugal AI Agent

Why one scene, two heads

The what is a shared explicit representation: a cloud of anisotropic 3D Gaussians, each with a mean, covariance, opacity, and appearance features, exactly the primitive of Section 51.2, optimized so that both sensors reproduce their real measurements. The why is consistency and cost. A camera constrains a Gaussian's color and its silhouette but is nearly blind to absolute depth; a lidar constrains that same Gaussian's range to centimeters but says little about color. Optimizing them jointly lets each sensor supply what it measures well and correct what the other measures poorly. A camera-only splat can float geometry anywhere along a ray and still look photometrically perfect; the lidar term nails it in place. The result is one geometry that is simultaneously photorealistic and metrically correct, which is precisely what a re-simulation pipeline (Section 51.3) needs to be trustworthy.

The how is that the two heads share the scene but not the projection. The camera head is standard perspective splatting. The lidar head cannot be: a spinning lidar does not image through a pinhole, it sweeps a set of laser beams across azimuth and reports range per return. So the lidar head projects each Gaussian into spherical coordinates, azimuth and elevation, and rasterizes onto the sensor's beam grid, compositing not color but expected range, intensity, and a probability that the ray returns at all. The when to reach for this over single-sensor splatting is any application that will consume both modalities downstream: autonomous-driving stacks, robot perception that fuses lidar and vision, or any augmentation set feeding a multimodal detector like those in Chapter 50.

Rolling shutter and the moving-platform problem

On a car at 15 m/s, the assumption that a frame is captured at one instant is simply false, and pretending otherwise bakes a ghost into the reconstruction. A rolling-shutter camera reads pixel rows sequentially over the exposure, so the bottom of the image is sampled milliseconds after the top. A spinning lidar is even more extreme: a full sweep takes about 100 ms, and every point in it is stamped at a different time as the beam rotates and the vehicle moves beneath it. If you splat the whole sweep from a single pose, straight lane markings bend and poles lean.

The fix is to make time a first-class coordinate of rendering. Each output element, a camera row or a lidar beam column, carries its own timestamp \(t\), and the sensor pose used to project the Gaussians is the interpolated ego pose at that exact \(t\). Concretely, if the ego trajectory is \(T(t) \in SE(3)\), the render for a pixel or beam fired at time \(t\) uses \(T(t)\), not the frame's nominal timestamp. Dynamic actors get the same treatment: each moving object has its own rigid trajectory, so a Gaussian belonging to a passing cyclist is placed at that actor's pose at \(t\) before projection. SplatAD folds this per-element timing into the rasterizer itself so that both sensors are unrolled correctly and remain mutually consistent, which is exactly why the joint formulation matters here, the timing model is shared rather than reinvented per sensor.

Rendering the lidar head: range, intensity, and ray drop

A lidar return is not one number. For each beam SplatAD predicts three things by alpha-compositing the Gaussians that fall on that beam, sorted front to back just as in camera splatting: the expected range (depth to first solid return), the intensity of the return (a learned per-Gaussian feature, since reflectivity depends on material and angle, see Chapter 2), and a ray-drop probability. That last term is what separates a convincing synthetic sweep from an obviously fake one: real lidars fail to get a return on dark, wet, specular, or grazing-angle surfaces, and the pattern of missing points is a strong tell. Rather than hand-code it, the model learns a per-ray dropout that a small network refines from the composited features, so the synthetic sweep has the same holes a real one would.

Training minimizes a sum of per-sensor losses on the raw measurements: a photometric loss on the camera image, an L1 (or L2) loss on lidar range, a loss on intensity, and a binary loss on ray drop, all evaluated against the recorded sensor data. Because everything is differentiable through the shared Gaussians, a range error at one beam nudges the same Gaussian that a color error at an overlapping pixel nudges. On the leakage-safe discipline running through this book, held-out timestamps or held-out camera rigs must define the evaluation split; scoring re-rendered frames at poses the optimizer already fit is measuring memorization, not synthesis, a point Section 51.7 makes into a full validation protocol.

import torch

def project_to_lidar_beams(mu_world, ego_pose_at_t):
    """Project Gaussian centers into a spinning-lidar's spherical frame.
    mu_world:        (N,3) Gaussian means in world coordinates
    ego_pose_at_t:   (N,4,4) the ego pose sampled at EACH point's own
                     firing time (rolling shutter / sweep timing)."""
    # Bring each point into the sensor frame using ITS timestamp's pose.
    ones = torch.ones(mu_world.shape[0], 1)
    homog = torch.cat([mu_world, ones], dim=1).unsqueeze(-1)      # (N,4,1)
    p_sensor = torch.matmul(torch.inverse(ego_pose_at_t), homog)  # (N,4,1)
    x, y, z = p_sensor[:, 0, 0], p_sensor[:, 1, 0], p_sensor[:, 2, 0]

    rng       = torch.sqrt(x*x + y*y + z*z)          # depth = lidar "pixel value"
    azimuth   = torch.atan2(y, x)                    # horizontal beam angle
    elevation = torch.asin(z / (rng + 1e-9))         # selects the beam row
    return azimuth, elevation, rng                   # rasterize onto beam grid

What joint splatting buys, and what it costs

The concrete gains are three. First, metric-photometric agreement: the synthetic camera and lidar cannot contradict each other because they read one geometry, which is the whole reason multimodal detectors trained on the output do not collapse. Second, speed: splatting rasterizes rather than ray-marches a network, so both sensors render at interactive-to-real-time rates where NeRF-based joint re-simulation ran seconds per frame, the difference between augmenting a dataset overnight and over a month. Third, editability: because actors are separate Gaussian sets with their own trajectories, you can move, add, or remove them and re-render both sensors coherently, which is the machinery Section 51.5 uses to manufacture rare events.

The costs are real and worth naming. Joint fitting needs accurate cross-sensor calibration and ego poses; a few centimeters of lidar-to-camera miscalibration puts the two loss terms in disagreement and smears the geometry. The scene is still a reconstruction of one drive, not a general world model, so extrapolating far from the recorded trajectory degrades. And appearance is baked at capture time, relighting to a different sun angle or weather is not free. These are exactly the seams that Section 51.7 teaches you to probe before trusting synthetic-from-real data in an evaluation loop.

What's Next

In Section 51.5, we put the editability of this joint scene to work: by relocating actors, inserting new agents, and perturbing trajectories, we generate synthetic examples of the rare and dangerous events that real logs almost never contain, and we ask what makes such synthetic data safe to train and evaluate on.