Diffusion-based depth (Marigold)

"Every other depth net was trained to answer. This one was trained to imagine, and it turns out that a machine that has imagined a billion scenes already knows roughly how far away things are."

A Generative AI Agent

The core trick: depth as a latent image

What Marigold (Ke et al., CVPR 2024) does is take a frozen pretrained Stable Diffusion v2 and fine-tune it into a depth estimator with minimal surgery. Stable Diffusion already contains two reusable parts: a variational autoencoder (VAE) that compresses a \(512\times512\) image into a small latent grid, and a denoising U-Net that runs the diffusion process in that latent space. Marigold keeps both. How it conditions on the input photograph is the whole idea: encode the RGB image to its latent \(\mathbf{z}^{(x)}\) with the frozen VAE, and at every denoising step concatenate that clean image latent with the current noisy depth latent \(\mathbf{z}^{(d)}_t\) along the channel axis. The U-Net therefore always sees the image it must explain while it denoises the depth. The generative process is the standard reverse diffusion,

started from pure Gaussian noise and run down to a clean depth latent that the VAE decoder turns back into a depth map. Why depth can ride through an image VAE at all is a small trick: depth is a single channel, so Marigold replicates it into three channels, normalizes it to the VAE's expected range, and treats it exactly like a grayscale picture. Remarkably, the image VAE reconstructs depth maps faithfully without retraining.

When this becomes practical is thanks to how little data it needs. The fine-tuning set is roughly seventy-four thousand samples drawn entirely from two synthetic datasets, Hypersim (indoor) and Virtual KITTI 2 (driving), where perfect dense ground-truth depth is free. No real depth sensor, no lidar scans, no messy captured data. Because the pretrained prior already knows what the world looks like, a few days of fine-tuning on synthetic geometry is enough to generalize zero-shot to real photographs the model never saw, echoing the leakage-safe, train-on-synthetic discipline this book keeps returning to.

The catch, and how the field killed it

The original recipe has an obvious weakness: reverse diffusion is iterative. Getting a good map meant running the U-Net for many denoising steps and, for the uncertainty ensemble, repeating that whole loop ten times. That is seconds per image on a strong GPU, hopeless for a robot or a wearable that needs frames now. Three lines of work collapsed this cost, and they matter because they turn a research curiosity into a deployable sensor.

First, consistency distillation: Marigold-LCM applies latent-consistency distillation so a usable depth map appears in one to four steps instead of dozens, a roughly ten-fold speedup with little quality loss. Second, and more surprising, end-to-end single-step fine-tuning: the follow-up work "Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think" (Martin Garcia et al., 2024) traced a subtle inference bug in the original pipeline and showed that once fixed, a single deterministic forward pass matches or beats the multi-step sampler. The diffusion machinery was, in the end, a very good initialization rather than a process you must run to completion. Third, flow matching: DepthFM (Gui et al., 2024) trains a straight-line transport from image to depth that likewise needs very few function evaluations. The through-line is that the geometric prior lived in the pretrained weights all along; the many-step stochastic sampler was never the point.

The pipeline in the figure makes the reuse explicit: the two VAE blocks are lifted unchanged from Stable Diffusion, and only the U-Net learns to speak depth.

Running it, and when to reach for it

Because Marigold ships in the Hugging Face diffusers library as a first-class pipeline, using it is short. The snippet below loads the distilled latent-consistency checkpoint, runs it in a handful of steps, and asks for a small ensemble so we also get an uncertainty map.

import torch
import diffusers

# Distilled few-step checkpoint from the ETH Photogrammetry group.
pipe = diffusers.MarigoldDepthPipeline.from_pretrained(
    "prs-eth/marigold-depth-lcm-v1-0",
    variant="fp16", torch_dtype=torch.float16,
).to("cuda")

image = diffusers.utils.load_image("scene.jpg")

out = pipe(
    image,
    num_inference_steps=4,      # LCM: a few steps, not dozens
    ensemble_size=5,            # resample to estimate uncertainty
    output_uncertainty=True,
)

depth = out.prediction          # affine-invariant depth, HxW in [0, 1]
sigma = out.uncertainty         # per-pixel std across the ensemble
depth_png = pipe.image_processor.visualize_depth(depth)[0]
depth_png.save("depth.png")

The output is affine-invariant, so it is ideal for tasks that care about relative geometry (matting, relighting, novel-view synthesis, obstacle ordering) and must be anchored before it can report meters. When to choose diffusion depth over the regressors of the previous sections comes down to three questions. Do crisp boundaries and thin structures matter more than raw throughput? Do you need calibrated-ish uncertainty without training an ensemble of networks? Is your domain far from any real depth dataset, so a strong pretrained prior beats a supervised regressor? If yes, Marigold-family models are the right tool. If you need metric depth out of the box or the tightest latency on a microcontroller, the metric and distilled feed-forward models of Section 41.3 remain the safer default.

What's Next

In Section 41.5, we confront the one thing every model in this chapter so far has refused to give us: absolute scale. We look at sensor-prompted depth, where a handful of sparse lidar or radar returns are fed into the network as a prompt, turning a beautiful but scale-free relative map into metric depth you can actually navigate by.