Stereo, structured light, and time-of-flight

"A camera hands me a flat picture and asks how far away everything is. The picture already threw that number away. My whole job is to find a second clue that puts it back."

A Depth-Perceiving AI Agent

Stereo: two eyes and a triangle

What stereo does is copy biology. Place two calibrated cameras a known distance apart, the baseline \(B\), pointing the same way. A scene point projects to a slightly different horizontal position in each image, and that shift, the disparity \(d\), encodes depth. Why it works is pure similar triangles: for cameras with focal length \(f\) (in pixels), the metric depth \(Z\) of a matched point obeys

Nearby objects shift a lot (large disparity), distant objects barely shift, and a point at infinity has zero disparity. How a stereo system spends its compute is almost entirely on the correspondence problem: for each pixel in the left image, find the matching pixel in the right. This is hard exactly where the scene is uninformative. A blank white wall, a repetitive brick facade, or a specular highlight gives the matcher no unique feature to lock onto, so disparity is undefined and depth is missing. Stereo is passive: it emits nothing, works outdoors in sunlight (where active infrared methods drown), scales to any baseline, and costs two ordinary image sensors. Its price is that it fails on textureless and repetitive surfaces, and its precision degrades with distance in a way the depth equation makes exact.

Structured light: paint the texture the scene forgot to have

Structured light keeps the triangulation geometry of stereo but fixes its worst failure. What changes is that one camera is replaced by a projector that casts a known pattern, stripes, dots, or a coded sequence, onto the scene. Why this helps is that the projector manufactures texture on surfaces that had none. The blank wall that defeated passive stereo now wears a crisp grid of infrared dots, and every dot is a feature the camera can locate. Because the projected pattern is known in advance, the correspondence problem collapses from an open-ended search into decoding: the camera reads which part of the pattern it sees and triangulates against the projector's known geometry. How the pattern is engineered defines the tradeoff. A single fixed speckle pattern (the approach of the original Kinect) works on moving scenes but gives modest resolution; a temporal sequence of shifting stripes (Gray-code or phase-shift profilometry) yields sub-millimeter accuracy but needs several frames of a static object, which is why industrial 3D scanners ask the part to hold still.

When structured light is the right tool is short-range, controlled-lighting, high-accuracy work: face unlock on a phone, a bin-picking robot inspecting parts at 0.5 m, a dental scanner. Its enemy is ambient infrared: bright sunlight swamps the projected pattern, so most structured-light depth cameras are indoor devices. It also cannot see past the range where the pattern is still resolvable, typically a few meters.

Figure 40.1.1 puts the families side by side so the shared idea is visible: both stereo and structured light are triangulation and obey \(Z = fB/d\), differing only in how they solve correspondence, while time of flight abandons geometry entirely for a clock.

Time of flight: measure the clock, not the angle

What a time-of-flight (ToF) sensor measures is how long light takes to fly to a surface and back. Since light travels at a known speed \(c\), a round-trip time \(t\) gives distance directly:

Why this is remarkable is the timescale: light covers 1 m round-trip in about 6.7 nanoseconds, so resolving centimeters demands picosecond timing. Two engineering answers exist. Direct ToF (dToF) fires a short laser pulse and times the returning photon with a fast detector such as a single-photon avalanche diode (SPAD); this is the principle of the lidar in Chapter 42 and of the ranging sensors now embedded in phones. Indirect ToF (iToF) sidesteps timing individual photons: it modulates the light source sinusoidally at frequency \(f_{\text{mod}}\) and measures the phase shift \(\varphi\) of the returned light per pixel, from which

How this shapes behavior is instructive. Phase wraps at \(2\pi\), so a single-frequency iToF camera has an unambiguous range of \(c / (2 f_{\text{mod}})\); a 50 MHz modulation wraps at 3 m, and a surface at 4 m reports 1 m. Multi-frequency schemes unwrap this the same way the phase-based methods of Chapter 3 resolve aliasing. The decisive property of all ToF is that depth error is roughly independent of range (limited by timing jitter and returned photon count), not quadratic like triangulation. That is why ToF and lidar own the automotive and long-range robotics regime while triangulation owns the near field. ToF's characteristic failure modes are multipath (light bouncing off two surfaces before returning, corrupting corners and shiny floors) and motion blur from the multiple sub-frames iToF integrates.

Reading it off in code, and choosing among the three

The stereo depth equation is concrete enough to compute directly. The snippet below takes a calibrated rig's parameters and a measured disparity map and returns metric depth, then shows the quadratic error growth that decides where stereo is trustworthy.

import numpy as np

def disparity_to_depth(disparity_px, focal_px, baseline_m):
    """Convert a disparity map (pixels) to metric depth Z = f*B/d."""
    d = np.asarray(disparity_px, dtype=float)
    Z = np.full_like(d, np.inf)          # zero disparity -> point at infinity
    valid = d > 0
    Z[valid] = focal_px * baseline_m / d[valid]
    return Z

f_px, B_m = 800.0, 0.12                  # 800 px focal length, 12 cm baseline
for Z in [1.0, 5.0, 20.0]:
    d = f_px * B_m / Z                    # disparity this depth produces
    dZ = Z**2 / (f_px * B_m) * 0.25       # depth error for 0.25 px disparity noise
    print(f"Z={Z:5.1f} m  disparity={d:6.2f} px  depth_error~{dZ:6.3f} m")
# Z=  1.0 m  disparity= 96.00 px  depth_error~ 0.003 m
# Z=  5.0 m  disparity= 19.20 px  depth_error~ 0.065 m
# Z= 20.0 m  disparity=  4.80 px  depth_error~ 1.042 m

Code 40.1.1 computes disparity-to-depth by hand so the geometry is transparent, but note it takes the disparity map as given. Producing that map, the actual correspondence search, is where a production stereo pipeline spends its effort, and a library makes it trivial.

The three methods are not competitors so much as tenants of different regimes, and one table settles most design arguments. Passive stereo: emits nothing, thrives in sunlight, fails on textureless and repetitive surfaces, error \(\propto Z^{2}\), best from tens of centimeters to a few meters. Structured light: active near-field triangulation, superb sub-millimeter accuracy indoors, defeated by sunlight and range, ideal under 2 m. Time of flight: dense per-pixel range from a clock, error roughly flat with range, troubled by multipath and reflective surfaces, and the only one of the three that reaches automotive distances. Neural methods that estimate depth from a single image, sidestepping all three physical tricks by learning scene priors instead, are the subject of Chapter 41, and they increasingly complement these hardware sensors rather than replace them.

What's Next

In Section 40.2, we take the per-pixel range these three sensors produce and turn it into the two data structures the rest of Part IX consumes: the depth map, a range image aligned to the camera, and the point cloud, an unstructured set of 3D coordinates obtained by back-projecting every valid depth through the calibration. That conversion is where the geometry of this section becomes the substrate for detection, registration, and the 3D representations of Chapter 42.