Depth maps and point clouds

"A depth map is a photograph that remembers how far away everything was. A point cloud is the same scene after it stopped caring where the camera stood."

A Reprojecting AI Agent

This section assumes you are comfortable with the pinhole camera model and camera intrinsics from Chapter 2. We treat calibration and the full menagerie of coordinate frames as given here and defer their rigorous treatment to Section 40.3. Our job in this section is narrow and load-bearing: define both representations precisely, derive the map that connects them, and catalog the encoding traps that silently corrupt real pipelines.

The depth map: a 2.5D image

A depth map \(D\) is a single-channel image the same size as the intensity image it accompanies. The value \(D(u,v)\) at pixel column \(u\), row \(v\) is a distance in physical units. Two conventions coexist and are routinely confused. Z-depth stores the perpendicular distance from the camera plane along the optical axis, the \(Z\) coordinate of the surface point. Range (or radial depth) stores the Euclidean distance from the camera center to the surface point along the ray. They differ by the ray's obliquity: a point at the image edge has larger range than Z for the same physical surface. Most RGB-D cameras and depth-estimation networks emit Z-depth; lidar and some time-of-flight sensors report range. Assuming the wrong one skews your reconstruction outward toward the frame edges.

We call a depth map "2.5D" because it captures the visible front surface of the scene from one viewpoint and nothing behind it. It is a function over the image grid, so it inherits every convenience of images: it is dense, it aligns pixel-for-pixel with color for easy fusion, and it feeds straight into a 2D convolutional network. Those properties are exactly why monocular depth foundation models in Chapter 41 predict depth maps rather than point clouds: the output stays on a regular grid a CNN or transformer can regress.

The point cloud: geometry without a grid

A point cloud \(P = \{\mathbf{p}_i\}\) is a set of 3D points \(\mathbf{p}_i = (x_i, y_i, z_i)\), optionally carrying per-point attributes: RGB color, surface normal, lidar return intensity, semantic label, or timestamp. The set has no intrinsic ordering and no fixed size; two frames of the same scene may hold wildly different point counts. This is the format that lidar produces natively and the format the detection and segmentation backbones of Chapter 42 consume.

A useful distinction is organized versus unorganized clouds. An organized cloud, produced by unprojecting a depth map, retains the \(H \times W\) image layout: point \((u,v)\) sits at a known array index, so pixel neighborhoods map to spatial neighborhoods for free and normal estimation is cheap. An unorganized cloud, typical of a spinning lidar merged across beams, is a flat list with no such structure; finding a point's neighbors requires a spatial index such as a k-d tree. Preserving organization when you have it saves enormous downstream cost, a theme we develop in Section 40.4.

Unprojection: from pixels to points

The bridge is the pinhole model. Given intrinsics with focal lengths \(f_x, f_y\) and principal point \((c_x, c_y)\), a pixel \((u,v)\) with Z-depth \(d = D(u,v)\) unprojects to the camera-frame point

Run this over every valid pixel and you have an organized point cloud in the camera frame. The inverse, projecting a point back to a pixel, divides by \(z\) and is how you would render a cloud into a new view. Two subtleties bite here. First, invalid measurements (out of range, low confidence, or occluded) are usually stored as \(0\) or NaN; unproject those blindly and you get a spurious wall of points at the origin. Second, if your depth is range rather than Z-depth, you must first convert: \(z = d / \sqrt{1 + ((u-c_x)/f_x)^2 + ((v-c_y)/f_y)^2}\) before applying the formulas above.

import numpy as np

def depth_to_points(depth_m, fx, fy, cx, cy, min_d=0.2, max_d=8.0):
    """Unproject a Z-depth map (meters) to an organized camera-frame cloud."""
    H, W = depth_m.shape
    u, v = np.meshgrid(np.arange(W), np.arange(H))
    z = depth_m
    x = (u - cx) * z / fx
    y = (v - cy) * z / fy
    pts = np.stack([x, y, z], axis=-1)          # (H, W, 3), stays organized
    valid = (z > min_d) & (z < max_d) & np.isfinite(z)
    return pts, valid                            # mask, do not delete, to keep the grid

# 16-bit PNG depth is almost always millimeters: convert before use.
raw = np.random.randint(0, 4000, (480, 640), dtype=np.uint16)  # stand-in frame
depth_m = raw.astype(np.float32) / 1000.0                      # mm -> m
pts, valid = depth_to_points(depth_m, fx=525., fy=525., cx=319.5, cy=239.5)
print(pts.shape, "valid points:", int(valid.sum()))

The code above makes the unprojection concrete and, more importantly, exposes the two failure points in one place: the unit scale on the raw PNG and the validity mask. Both are invisible in the math and unforgiving in practice.

Choosing a representation

Prefer the depth map when the task lives on the image grid: learned depth regression, dense fusion with color, or any 2D-convolutional stage. Prefer the point cloud when geometry is the point: merging views across a moving camera, registering scans, fitting planes and objects, or feeding a 3D detection backbone. Many systems keep both and convert lazily. A SLAM front end (Chapter 52) may track on depth maps for speed while accumulating a global point cloud map for loop closure. The uncertainty attached to each depth value, which you will learn to propagate in Section 40.6, follows the measurement through the unprojection and should travel with the points.

What's Next

In Section 40.3, we make the coordinate frames we glossed over here fully explicit: how intrinsics are calibrated, how extrinsics relate the depth sensor to the color camera, the IMU, and the robot base, and how a clean chain of transforms turns a camera-frame cloud into a world-frame map you can trust.