Using depth FMs inside a sensing stack

"A benchmark rewards the sharpest map. A robot rewards the map that arrives on time, admits when it is guessing, and never tells the planner the wall is a doorway."

A Systems-Minded AI Agent

This section is the integration capstone of the chapter, so it assumes the whole chapter behind it: affine-invariant versus metric prediction from Section 41.1 and Section 41.3, sensor prompting from Section 41.5, and the evaluation discipline of Section 41.6. It also reaches forward to the consumers of depth: bird's-eye-view and occupancy in Chapter 43, spatial mapping in Chapter 52, and the fusion machinery of Chapter 48.

What role does the depth FM play?

Decide the job before the model. A depth FM can sit in a stack in three distinct roles, and the right engineering follows from which one you pick. As a geometry primitive, it turns each frame into a point cloud that feeds detection, occupancy, or obstacle avoidance; here density and edge accuracy matter most, and scale must be sourced somewhere. As a prior or initializer, it seeds a downstream optimizer: dense monocular depth is a superb initialization for multi-view stereo or for the geometry term in a SLAM back end, which then refines it with true triangulation. As a gap-filler, it densifies a real but sparse sensor, covering the returns lidar misses on dark or specular surfaces. The role dictates the scale strategy. A primitive that drives a metric planner needs metric depth, so you either run a camera-agnostic metric model or prompt an affine model with a sparse sensor. A prior that will be triangulated later can stay relative, because the downstream optimizer supplies scale from motion. Confusing these is the most common integration mistake: teams demand metric perfection from a model whose output was going to be re-scaled anyway, or ship a relative model into a planner that silently assumed meters.

Latency, resolution, and the edge budget

A depth FM that runs at 2 frames per second on a workstation GPU is a research artifact, not a perception module. Real stacks impose a latency budget, and depth usually shares a compute envelope with detection, tracking, and planning. Three levers bring a large depth transformer into budget. First, resolution: inference cost scales with pixel count, and most planners need accurate geometry near obstacles more than crisp far detail, so running at a reduced resolution and upsampling with edge guidance often costs little accuracy where it matters. Second, backbone choice and distillation: model families ship small, base, and large variants, and a distilled small student trained from a large teacher recovers most of the accuracy at a fraction of the FLOPs. Third, quantization and compilation: converting to INT8 or FP16 and compiling to the target runtime (TensorRT, ONNX Runtime, Core ML, a mobile NPU) is the difference between a demo and an on-device feature, the toolkit of Chapter 59. Measure end to end: preprocessing and the sparse-alignment fit and the lift to a point cloud all consume the budget, and a model that is fast in isolation can miss the deadline once wrapped.

Video adds a second, subtler cost: temporal stability. Run a per-frame image model on a video and the depth flickers, because tiny input changes move the prediction and there is no constraint tying consecutive frames. A planner reading that stream sees phantom motion. Video depth models add temporal attention or a consistency loss so the geometry holds still when the scene does, and a lightweight alternative is to smooth per-frame depth through the pose from your odometry, warping the previous frame's depth into the current view and blending. Either way, temporal consistency is a hard requirement for control, not a nicety.

import numpy as np

def depth_fm_stack_step(rgb, sparse_z, sparse_mask, K, prev_depth=None,
                        flow_warp=None, conf_thresh=0.15):
    """One frame of a depth-FM perception step: predict, scale, gate, lift.

    rgb        : (H, W, 3) current camera frame
    sparse_z   : (H, W) metric returns from a range sensor (lidar / ToF)
    sparse_mask: (H, W) bool, True where the range sensor has a return
    K          : (3, 3) camera intrinsics, for lifting depth to a point cloud
    prev_depth : (H, W) previous metric depth, in the current view (pose-warped)
    """
    disp = run_depth_fm(rgb)                       # affine-invariant inverse depth
    a, b = fit_scale_shift(disp, sparse_z, sparse_mask)   # meters from the sensor
    depth = 1.0 / np.clip(a * disp + b, 1e-6, None)       # dense metric depth

    if prev_depth is not None:                     # temporal smoothing for control
        depth = 0.7 * depth + 0.3 * prev_depth     # blend warped previous frame

    # Confidence from agreement with the sparse sensor at held-out returns.
    resid = np.abs(depth[sparse_mask] - sparse_z[sparse_mask]) / sparse_z[sparse_mask]
    frame_conf = float(np.median(resid))
    trustworthy = frame_conf < conf_thresh         # gate before the planner sees it

    points = lift_to_pointcloud(depth, K) if trustworthy else None
    return depth, points, trustworthy, frame_conf

Uncertainty, failure modes, and safety gating

A depth FM fails in ways that are quiet and geometry-shaped, which makes them dangerous. It hallucinates plausible depth on a blank wall, flattens a reflective floor into an abyss, reads a printed poster of a hallway as a real hallway, and collapses in scenes far from its training distribution. None of these announce themselves; the output looks like a confident, smooth depth map. So a depth FM must never feed a safety-critical planner as ground truth. Wrap it in three guards. First, a confidence signal: the cleanest source is disagreement with an independent measurement, the residual against held-out sparse returns in the code above, which is why keeping even a cheap range sensor in the loop pays off twice. Second, calibrated uncertainty: rather than trust a raw confidence, calibrate depth intervals so that a stated 90 percent interval contains the true depth 90 percent of the time, the conformal machinery of Chapter 18. Third, a fallback: when the gate trips, degrade to the raw range sensor, hold the last trusted map, or slow the system down, rather than acting on a hallucinated hallway. Adversarial framing matters too, because a printed scene or a projected pattern can spoof a monocular model in ways it cannot spoof a time-of-flight sensor, the threat model of Chapter 68.

When not to reach for a depth FM

The mark of maturity is knowing when the answer is no. If you need certified metric accuracy for a safety stop, a real range sensor with a physical measurement principle is the right tool, and a monocular estimate is at best a redundant cross-check, developed as a modality in Chapter 42. If the scene is textureless, transparent, or specular, the monocular prior degrades exactly where a time-of-flight or structured-light sensor still works, though even those struggle on glass. If a fixed, calibrated stereo rig already covers the field of view, its triangulated metric depth is cheaper to trust than a learned prior. And if the platform cannot spare the compute after quantization, a classical stereo or a cheap ToF module beats a depth FM that misses the deadline. The depth FM shines when a camera is already present, when dense geometry is needed on surfaces a sparse sensor misses, when the deployment spans domains no single dataset covered, and when the consumer of depth can tolerate a gated, uncertainty-aware estimate rather than a certified measurement. Treat it as one strong, cheap, general-purpose estimator among several, fused rather than trusted, and the stack gets the best of the foundation-model prior without inheriting its failure modes.

What's Next

In Chapter 42, we turn to the real range sensor that has anchored every scale prompt and safety gate in this chapter: lidar. We build the point-cloud representations, sparse-convolution and transformer backbones, and detection and segmentation pipelines that let a machine reason directly in metric 3D, the measurement that a monocular depth FM approximates and a certified stack still depends on.