"Give me sixty million photos nobody bothered to label and I will tell you how far away everything is. Ask me for the answer in meters and we should have a longer conversation."
A Self-Taught AI Agent
Prerequisites
This section builds directly on Section 41.1, where the shift from per-dataset depth regression to zero-shot relative depth, and the scale-and-shift-invariant loss that makes it possible, are developed. It assumes the depth geometry and the relative-versus-metric distinction of Chapter 40, and the self-supervised pretraining ideas (DINOv2-style visual encoders) from Chapter 17. The zero-shot transfer language recurs formally in Chapter 66.
The Big Picture
Depth Anything is the model that made monocular depth feel solved for everyday use: point any camera at any scene, run one forward pass, and get a clean depth map back, no calibration, no stereo rig, no per-scene tuning. It got there not through a cleverer architecture but through a data engine: a strong teacher network was used to pseudo-label tens of millions of ordinary unlabeled photos, and a student was trained on that flood of automatically annotated data. Version 2 then quietly rebuilt the recipe around synthetic training images and produced markedly sharper, more reliable depth. This section explains how the data engine works, why v2's synthetic-teacher trick matters, what the model's output actually means (relative depth, not meters), and how the same idea was extended to video so that a walk-through clip does not flicker frame to frame.
The data engine: how v1 scaled without labels
The bottleneck for monocular depth was never the network; it was ground truth. Metric depth labels come from lidar, structured light, or stereo rigs, all of which are expensive, indoor-biased, sparse, and full of holes. Depth Anything v1 (Yang and colleagues, 2024) sidestepped the bottleneck with self-training at scale. First it trained a strong teacher on the roughly 1.5 million labeled images available across public datasets, using the scale-and-shift-invariant objective from Section 41.1 so that images from wildly different sensors could be pooled. Then it turned that teacher loose on about 62 million unlabeled internet images, producing a pseudo-label depth map for every one. Finally it trained a student on the combined pile, dominated by the 62 million pseudo-labeled frames. The encoder is a frozen-then-finetuned DINOv2 vision transformer; the decoder is the DPT dense-prediction head. Two design choices made the flood productive rather than merely large: the student was hit with strong perturbations (aggressive color jitter, blur, and CutMix) so it could not simply copy the teacher and instead had to learn robust cues, and an auxiliary feature-alignment loss tied the depth features to DINOv2's frozen semantic features, importing object knowledge that keeps the depth map from bleeding across object boundaries.
Key Insight
The leap in monocular depth came from data coverage, not model capacity. A teacher that is merely good, applied to 62 million diverse unlabeled images, manufactures a training set whose scene variety no hand-labeled depth dataset can match. Diversity of viewpoints, lighting, and object categories is what buys zero-shot generalization; the perturbations and the semantic-feature constraint are what stop the student from just memorizing the teacher's mistakes.
Depth Anything V2: let the teacher learn from synthetic data
Version 2 (2024) kept the data-engine skeleton but changed what the teacher learns from, and the change is the whole story. Real labeled depth is noisy: lidar misses thin structures, stereo smears edges, transparent and reflective surfaces are simply wrong. Synthetic renderings, by contrast, carry perfect, pixel-sharp, complete depth (every strand of hair, every mesh edge) but look subtly unreal. V2's recipe: train the teacher only on high-quality synthetic images so it inherits their razor-sharp geometry, scale that teacher up to the largest DINOv2-Giant encoder so it can bridge the synthetic-to-real appearance gap, then use it to pseudo-label a large pool of real unlabeled photos, and finally train the deployable students purely on those real pseudo-labels. The synthetic domain gap, usually a curse, is laundered through the pseudo-labeling step: students never see synthetic pixels, only real ones with clean synthetic-quality depth. The result is depth that is visibly crisper at boundaries and far more robust on hard cases (transparent objects, thin poles, reflections) than v1, while shipping in a family of sizes from a 25-million-parameter Small model, fast enough for a laptop CPU, up to the 1.3-billion-parameter Giant.
Research Frontier
As of 2026, Depth Anything V2 is the default open-weight backbone for zero-shot relative depth, and its published DINOv2-plus-DPT design is the reference point new work is measured against. The active frontiers are (1) turning its relative output into calibrated metric depth without per-camera tuning, handled by the camera-agnostic models of Section 41.3; (2) diffusion-based depth (Marigold, Section 41.4) that trades speed for detail; and (3) temporally consistent video depth, covered below, where 2025's Video Depth Anything extended the family to long clips.
What the output actually is, and how to use it
A crucial discipline: Depth Anything predicts relative (affine-invariant) depth, not metric depth. The network output is proportional to inverse depth (disparity) up to an unknown global scale \(a\) and shift \(b\); the true metric depth of a pixel relates to the prediction \(\hat{d}\) by
\[ \frac{1}{Z} \approx a\,\hat{d} + b, \qquad a, b \text{ unknown per image}. \]This means the map tells you correctly that the chair is nearer than the wall and roughly how much nearer, but it does not tell you either object is two meters away. To recover \(a\) and \(b\) you need at least a little metric anchoring: a couple of known-range points from a lidar return, a stereo baseline, or a known object size. Fit the two constants by least squares against those anchors and the whole relative map snaps to metric. That two-parameter fit is the seam where a depth foundation model plugs into a real sensing stack, and it is exactly the lidar-guided prompting theme of Section 41.5. Getting a relative depth map from an image is now a few lines, as Listing 41.2 shows.
from transformers import pipeline
from PIL import Image
# Zero-shot relative depth: one model handles any scene, no calibration.
depther = pipeline(task="depth-estimation",
model="depth-anything/Depth-Anything-V2-Small-hf")
image = Image.open("street.jpg")
out = depther(image)
depth = out["depth"] # PIL image; larger value = nearer (inverse depth)
# Relative -> metric needs anchoring. With a few (row, col, Z_true) lidar hits:
import numpy as np
d = np.asarray(depth, dtype=np.float32)
rows, cols, Z_true = anchors[:, 0].astype(int), anchors[:, 1].astype(int), anchors[:, 2]
A = np.stack([d[rows, cols], np.ones_like(Z_true)], axis=1)
a, b = np.linalg.lstsq(A, 1.0 / Z_true, rcond=None)[0] # fit inverse-depth affine
Z_metric = 1.0 / (a * d + b) # metric depth, meters
Listing 41.2 makes the division of labor explicit: the network solves the hard, ambiguous perception problem (dense ordering of the scene by depth), and a trivial two-parameter fit against any metric anchor handles the part cameras physically cannot infer alone. Keep the two stages separate in your head and the model stops feeling like a magic meter-stick and starts behaving like a well-characterized sensor.
The Right Tool
The from-scratch path, download the checkpoint, rebuild the DINOv2 encoder and DPT head, match the preprocessing (resize to a multiple of 14, normalize, letterbox), run inference, and resize back, is roughly 120 lines of fiddly code where one wrong normalization constant silently degrades the output. The Hugging Face transformers pipeline in Listing 41.2 collapses all of it to three lines and pins the exact preprocessing the weights were trained with, so the only thing you can get wrong is the file path.
Video depth: killing the flicker
Run a per-frame depth model on a video and you get a subtler failure than any single frame reveals: temporal flicker. Because each frame is estimated independently, the unknown scale and shift wander from frame to frame, and a static wall breathes in and out by meters while the camera pans. For a still photo this is invisible; for a robot building a map or an AR app anchoring a virtual object, it is fatal, because the geometry it is fusing over time is inconsistent with itself. Video Depth Anything (2025), from the same group, extends V2 to clips by adding a lightweight spatiotemporal head over the frozen image backbone and training with a temporal-gradient consistency loss that penalizes depth changes between consecutive frames that are not explained by actual motion. Critically it does this without a global optimization over the whole clip, so it streams: it processes arbitrarily long videos (thousands of frames) at near-real-time on the smaller sizes, keeping depth stable over time while preserving V2's per-frame sharpness. This temporal-consistency problem, and the gradient-style loss that tames it, is the same shape as the sequence-smoothing ideas in Chapter 14, applied to a stream of depth maps instead of a stream of scalars.
In Practice: AR furniture placement on a phone
A home-furnishing app lets a shopper drop a virtual sofa into their living room through the phone camera. The team first tried per-frame Depth Anything V2 for occlusion (so a real coffee table correctly hides the sofa's front leg). It looked stunning in screenshots and awful in motion: as the user walked around the sofa, the table's estimated depth jittered, so the occlusion boundary shimmered and the sofa's leg flickered in and out from behind it. Switching to Video Depth Anything's streaming, temporally consistent depth removed the shimmer: the table's depth now held steady across frames, the occlusion edge stayed put, and the sofa sat convincingly on the floor. The lesson generalizes to any moving-camera use, robot navigation, drone mapping, wearable AR: for video, temporal consistency is not a polish item, it is the difference between usable and unusable depth. It is also why SLAM and spatial-AI stacks increasingly ingest video depth models rather than per-frame ones.
Exercise
Using Listing 41.2 as a starting point: (a) run Depth Anything V2 Small on three photos of the same room from different distances and confirm that the raw relative maps are not comparable in absolute value across images. (b) Simulate four lidar anchor points per image by reading off known distances, fit \((a, b)\) for each, and check whether a single shared \((a, b)\) works across all three or whether each image genuinely needs its own fit. (c) Explain, in terms of the affine ambiguity, why a video model that keeps \((a, b)\) consistent over time is doing something a per-frame model structurally cannot.
Self-Check
1. Depth Anything v1 was trained mostly on data it never had human labels for. Where did the labels come from, and what two tricks stopped the student from simply copying the teacher's errors?
2. Depth Anything V2 trains its teacher only on synthetic images yet its deployed students never see a synthetic pixel. How does the pseudo-labeling step turn the synthetic-to-real domain gap from a liability into an advantage?
3. The model outputs relative depth. Write the affine relation to metric depth, say how many anchor points you minimally need to solve for the unknowns, and name one sensor that can supply them.
What's Next
In Section 41.3, we tackle the constant we kept deferring: true metric depth in meters, straight from a single image, without the per-image affine fit. We look at camera-agnostic models (Metric3D v2, UniDepth) that fold the camera's intrinsics into the network so the output is already scaled, and we see why doing this correctly is harder, and more valuable, than it first appears.