Evaluation and zero-shot transfer

"Tell me which datasets you never trained on, and I will tell you whether your depth model can see."

A Skeptical AI Agent

This section assumes you have met the models themselves (Section 41.2 through Section 41.5) and the geometry of depth from Chapter 40. It leans on two running threads of this book: leakage-safe evaluation, treated in general in Chapter 65, and distribution shift, treated in Chapter 66. The novelty here is the depth-specific twist: the prediction and the ground truth do not live in the same units.

Why depth evaluation needs alignment first

Most monocular foundation models predict affine-invariant depth or, more precisely, affine-invariant inverse depth (disparity). The network fixes the shape of the scene but leaves an unknown global scale \(s\) and shift \(t\). This is not sloppiness; it is what the data supports. A single image of a room and the same image of a doll house are pixel-identical, so no monocular model can recover absolute metric scale without a prior on the camera or the scene (that is exactly what the metric and camera-aware models of Section 41.3 add). Evaluation therefore has to remove the degrees of freedom the model was never asked to predict, or every metric collapses to "you guessed the wrong scale."

The standard fix is a least-squares scale-and-shift alignment, computed per image in inverse-depth space. Let \(d_i\) be predicted disparity and \(d^{*}_i = 1/z^{*}_i\) the ground-truth disparity at valid pixels. Solve

then compare the aligned prediction \(\hat{z}_i = 1/(s\,d_i + t)\) against \(z^{*}_i\). For models that claim metric output you skip the shift and use only a single scale (often the ratio of medians, which is robust to outliers). Aligning in inverse-depth space matters: it weights near geometry, where disparity is large, the way stereo and structure-from-motion errors actually behave.

The metrics that reward geometry

Once aligned, report a small standard panel rather than a single number, because different metrics catch different failures. The workhorses are absolute relative error and the threshold accuracies:

AbsRel is scale-relative, so a 10 cm error at 1 m is penalized like a 1 m error at 10 m, which matches how depth uncertainty grows with range. The \(\delta_1\) accuracy (the fraction of pixels within a 25 percent ratio band) is the headline number for zero-shot leaderboards because it is intuitive and bounded in \([0,1]\). Alongside them, the scale-invariant log error (SILog, from Eigen's early work) penalizes the relative structure of the depth map while ignoring global scale, and boundary or edge-accuracy metrics (F-score on depth discontinuities) expose the blurring that pixel-averaged errors hide. For video depth from Section 41.2, add a temporal metric such as optical-flow-based warping error or temporal alignment error, since a per-frame-accurate model can still flicker unusably.

The zero-shot cross-dataset protocol

Zero-shot transfer is the property that made these models a paradigm shift, so it needs its own protocol. The recipe, established by MiDaS and inherited by Depth Anything, Metric3D, and Marigold, is: train on a large mixture of datasets, then evaluate on a set of benchmark datasets that are held entirely out of training. The classic held-out suite spans indoor and outdoor, real and synthetic: NYU Depth v2 and ScanNet (indoor RGB-D), KITTI (driving lidar), ETH3D and DIODE (high-precision laser scans), and Sintel (synthetic film). Because none of these appear in training, a strong score is evidence of genuine transfer rather than memorization. This is a stricter bar than the per-dataset regime of Section 41.1, where train and test come from the same distribution and a model can overfit the sensor's quirks.

Two disciplines make or break the protocol. First, use each benchmark's own evaluation crop, cap, and mask; NYU uses the Eigen crop, KITTI uses the Garg crop, and comparing across inconsistent crops is a classic silent error. Second, guard against leakage, which for image data is subtle. The same building photographed for two datasets, frames sampled from a shared video source, or a synthetic asset reused across corpora can put near-duplicates on both sides of the train/test line. Perceptual near-duplicate detection over the training mixture and the held-out sets, in the spirit of the leakage-safe pipeline in Chapter 5, should be a gate, not an afterthought.

Running the evaluation yourself

The core of the protocol is short. The function below performs least-squares scale-and-shift alignment in disparity space and then computes AbsRel and \(\delta_1\) on the valid, capped pixels, which is exactly what a zero-shot leaderboard entry does per image before averaging.

import numpy as np

def align_and_score(pred_disp, gt_depth, mask, max_depth=80.0):
    """Affine-align predicted disparity to metric depth, then score."""
    valid = mask & (gt_depth > 1e-3) & (gt_depth < max_depth)
    gt_disp = 1.0 / gt_depth[valid]
    d = pred_disp[valid]

    # Least-squares fit of s, t so that s*d + t approximates gt_disp.
    A = np.stack([d, np.ones_like(d)], axis=1)
    s, t = np.linalg.lstsq(A, gt_disp, rcond=None)[0]

    pred_depth = 1.0 / np.clip(s * d + t, 1e-6, None)
    gt = gt_depth[valid]

    abs_rel = np.mean(np.abs(pred_depth - gt) / gt)
    ratio = np.maximum(pred_depth / gt, gt / pred_depth)
    delta1 = np.mean(ratio < 1.25)
    return {"AbsRel": float(abs_rel), "delta1": float(delta1)}

This is fine for understanding the mechanics, but reproducibility demands the exact per-benchmark crops, caps, and masks, which are easy to get subtly wrong by hand.

Calibration, uncertainty, and honest reporting

A depth number without an error bar is a liability in any control loop. Foundation models are typically overconfident on out-of-distribution inputs, so pair the accuracy panel with a calibration view: bin pixels by predicted confidence (or, for the diffusion models of Section 41.4, by the ensemble variance across samples) and check that stated uncertainty tracks realized error. The conformal-prediction machinery of Chapter 18 gives distribution-free depth intervals with a coverage guarantee, which is far more useful to a downstream planner than a bare point estimate. Report per-range and per-surface breakdowns as a matter of course, since global averages, as the warehouse story showed, launder the failures that matter most.

What's Next

In Section 41.7, we stop measuring depth foundation models in isolation and wire them into a working sensing stack: feeding aligned, uncertainty-tagged depth into fusion, SLAM, and planning, and deciding when a monocular prior is enough and when it must be anchored by lidar, radar, or a metric camera model.