3D semantic occupancy prediction (Occ3D, SurroundOcc, TPVFormer)

"A bounding box tells me a truck is there. Occupancy tells me the ladder sticking out of its bed is about to be there too."

A Cautious AI Agent

This section builds directly on the camera-to-BEV lifting of Section 43.2 and the multi-sensor BEV grids of Section 43.3; you should be comfortable with lift-splat-scatter and with deformable BEV attention before continuing. It also leans on the voxel and point-cloud machinery of Chapter 42, since lidar sweeps are what generate the ground-truth voxels here. Our narrow job: derive why a full dense 3D grid is expensive, present the two representational tricks (tri-perspective and coarse-to-fine) that tame that cost, and explain how the Occ3D labels are actually made so you can trust or distrust the numbers a benchmark reports.

From boxes to voxels: why occupancy is a different problem

Discretize a region of interest, say 40 meters ahead and behind, 40 meters to each side, and roughly 6 meters of height, into voxels of 0.4 meter. That is a grid of about \(200 \times 200 \times 16\), which is 640,000 cells. A detector predicts perhaps a few dozen boxes; an occupancy model must emit a semantic label for every one of those cells at every frame. The naive storage and compute both scale with the product of the three axes, and a dense 3D convolution over such a grid is the operation that made early occupancy networks too heavy for a car's compute budget. The payoff for spending that budget is open-set safety: because the model reasons about occupied volume rather than named instances, an obstacle from an unseen category still registers as occupied, so the planner can avoid a thing it cannot name. This is the property that a box detector structurally cannot offer.

Formally, let \(V \in \{0,1,\dots,K\}^{X \times Y \times Z}\) be the label volume over \(K\) semantic classes plus "empty." The model learns \(p(V \mid \mathcal{I})\) from surround images \(\mathcal{I}\), and training minimizes a voxel-wise loss. Because most of space is air, the class distribution is brutally imbalanced: often more than 90 percent of cells are empty. A plain cross-entropy will happily predict "empty everywhere" and score well, so occupancy models lean on class-balanced weighting and on the Lovász-softmax loss, which directly optimizes the intersection-over-union that the benchmark actually reports.

TPVFormer: three planes instead of one cube

The bird's-eye-view grid is one projection of 3D space onto the ground plane; it discards height. A full voxel grid keeps height but pays the cubic cost. TPVFormer (Huang et al., CVPR 2023) splits the difference with a tri-perspective view: instead of one BEV plane or one dense cube, it maintains three orthogonal feature planes, the top-down \(xy\), the front \(yz\), and the side \(xz\). Any point in space is described by summing the features sampled from its projection onto each of the three planes. This is a rank-decomposition of the volume: three 2D maps of size \(HW + WD + HD\) stand in for a cube of size \(HWD\), turning cubic storage into quadratic. Cross-attention lifts the surround-image features onto each plane using the same deformable-attention lifting you saw in Section 43.2, just applied three times with three sets of reference points.

To read the occupancy of an arbitrary voxel at \((x,y,z)\), TPVFormer gathers the feature at \((x,y)\) on the top plane, \((y,z)\) on the front plane, and \((x,z)\) on the side plane, sums them, and passes the result through a small multilayer perceptron that outputs the class. Because the query point is continuous, the representation is resolution-free: you can decode occupancy at any voxel size, or even at raw lidar points, without retraining. That is why TPVFormer, though trained only with sparse lidar-point supervision, can render a dense volume at inference.

SurroundOcc: coarse-to-fine dense supervision

TPVFormer proved cameras could predict occupancy, but sparse lidar-point supervision leaves most voxels unlabeled, so the predicted volume is holey. SurroundOcc (Wei et al., ICCV 2023) attacks both ends. On the architecture side it builds a genuine 3D volume but in a coarse-to-fine cascade: 2D image features are lifted into a low-resolution 3D grid, then upsampled and refined through successive 3D deconvolution stages, so the expensive high-resolution convolutions run only on an already-structured volume rather than from scratch. On the supervision side it contributes a pipeline for dense occupancy ground truth: multiple lidar sweeps across a whole clip are aggregated into one dense point cloud, static scene and moving objects are stitched with Poisson surface reconstruction to fill holes, and the result is voxelized into a dense labeled volume. Training against dense targets rather than sparse points is what lets SurroundOcc produce solid, watertight surfaces instead of a cloud of confident dots.

How the Occ3D labels are made, and why the mask matters

A benchmark is only as trustworthy as its ground truth, and dense 3D occupancy ground truth is manufactured, not measured. The Occ3D pipeline accumulates lidar across a sequence into a dense cloud, uses the dataset's 3D box annotations to separate dynamic objects from the static background so moving cars are not smeared into walls, voxelizes the union, and then, decisively, computes a camera-visibility mask by ray-casting from each camera through the grid. Voxels that no camera ray could reach, occluded behind a building or below the road surface, are flagged as unobserved and excluded from evaluation. Without that mask a camera model would be penalized for failing to hallucinate the inside of a solid wall, which is not a perception failure but a physics one. Understanding this construction tells you exactly what an Occ3D score does and does not certify: it rewards correct reconstruction of the visible occupied surface, not omniscience.

The snippet below builds a tiny occupancy grid the way the label pipeline does, aggregating points into voxels and marking a coarse visibility mask, so the shape of the supervision target is concrete rather than abstract.

import numpy as np

def voxelize(points, labels, bounds, vox=0.4):
    # points: (N,3) in meters; labels: (N,) semantic class ids
    lo = np.array(bounds[0]); hi = np.array(bounds[1])
    dims = np.ceil((hi - lo) / vox).astype(int)          # grid X,Y,Z
    grid = np.zeros(dims, dtype=np.int8)                 # 0 = empty
    idx = np.floor((points - lo) / vox).astype(int)
    keep = np.all((idx >= 0) & (idx < dims), axis=1)
    idx, lab = idx[keep], labels[keep]
    grid[idx[:, 0], idx[:, 1], idx[:, 2]] = lab          # last write wins
    # visibility: a column is observed up to its highest filled voxel
    observed = np.zeros(dims, dtype=bool)
    filled_z = np.where(grid != 0, np.arange(dims[2])[None, None, :], -1)
    top = filled_z.max(axis=2)                           # (X,Y) top surface
    for z in range(dims[2]):
        observed[:, :, z] = z <= np.maximum(top, 0)
    return grid, observed

pts = np.random.uniform([-4, -4, 0], [4, 4, 4], size=(2000, 3))
lab = np.random.randint(1, 6, size=2000)
grid, mask = voxelize(pts, lab, bounds=([-4, -4, 0], [4, 4, 4]))
print("grid", grid.shape, "occupied", int((grid != 0).sum()),
      "evaluated", int(mask.sum()))

What's Next

In Section 43.5, we drop the expensive dense supervision entirely. Self-supervised occupancy learns geometry from raw sensor streams via differentiable rendering, and Gaussian occupancy (GaussianOcc) replaces the voxel grid with a cloud of 3D Gaussians, borrowing the splatting machinery of Chapter 51 to get continuous, memory-light occupancy without ever building the cube this section fought so hard to shrink.