Camera-to-BEV lifting and BEVFormer-style attention

"A camera insists everything it sees lies on a ray. My job is to argue back, politely, about how far."

A Perspective-Weary AI Agent

This section assumes the shared bird's-eye-view representation motivated in Section 43.1: a metric \((x, y)\) grid, ego-centered, where geometry is Euclidean and fusion is easy. It also assumes the pinhole camera model and the intrinsic and extrinsic matrices from Chapter 2, the transformer and attention vocabulary of Chapter 15, and an appreciation, from Chapter 41, of just how ambiguous monocular depth really is. Our job is narrow: define the two lifting paradigms, derive their mechanics, and name the tradeoffs that decide which one you deploy.

The lifting problem: one pixel, a whole ray of candidates

Project a 3D point \(\mathbf{p} = (X, Y, Z)\) in the camera frame to a pixel with the intrinsic matrix \(K\):

The forward map is exact, but it destroys one number: the depth \(d\). Inverting it, going from pixel \((u, v)\) back to 3D, requires \(d\), and a single image simply does not contain it. Every pixel corresponds not to a point but to a ray, a one-parameter family of candidate positions. Camera-to-BEV lifting is, at heart, the discipline of resolving or hedging that missing depth so that image content can be placed on the ground plane. The two paradigms differ in exactly how they handle the ambiguity: one represents it explicitly as a distribution, the other sidesteps it by asking the grid what it wants.

Lift-Splat-Shoot: push pixels along a depth distribution

Lift-Splat-Shoot (LSS), introduced by Philion and Fidler in 2020, refuses to commit to a single depth. For each pixel it predicts a categorical distribution over a discrete set of depth bins \(\{d_1, \dots, d_D\}\), typically a few dozen bins spanning 1 to 60 meters. The image backbone emits, per pixel, both a context feature vector \(\mathbf{c} \in \mathbb{R}^{C}\) and a depth probability vector \(\boldsymbol{\alpha} \in \Delta^{D-1}\). The lift step forms a frustum of \(D\) points along the ray and weights the context feature by the depth probability at each bin:

This outer product is the trick. A confident pixel concentrates its feature at one depth; an ambiguous one smears it softly across several. No hard decision is ever made, so gradients flow and the network learns depth as a byproduct of the downstream task. The splat step then uses the known camera geometry to place each of these \(H \times W \times D\) frustum features at its 3D location, drops them into BEV grid cells, and sum-pools everything landing in the same cell, a scatter operation identical in spirit to the pillar scatter of Chapter 42. Multiple cameras splat into the same shared grid, which is why LSS handles a full surround rig natively. The shoot in the original name refers to a motion-planning head; in modern usage LSS means the lift-and-splat encoder.

BEVFormer: pull features into learnable BEV queries

BEVFormer (Li et al., 2022) inverts the data flow. Rather than pushing pixels out, it starts from the destination: a grid of \(H_{\text{bev}} \times W_{\text{bev}}\) learnable BEV queries, one per ground cell. Each query is responsible for gathering the image evidence relevant to its patch of the world, using two attention mechanisms.

Spatial cross-attention lets a BEV query at ground location \((x, y)\) reach into the camera images. Because the query knows its own \((x, y)\) and a small set of candidate heights (pillars of reference points), it uses the camera extrinsics and intrinsics to project those 3D reference points into whichever images actually see them, then samples features there. Crucially, BEVFormer uses deformable attention: each projected point attends to only a handful of learned offset locations around its projection, not the whole image, which keeps cost linear rather than quadratic in image size. This is the inverse of LSS: instead of every pixel guessing its depth, every BEV cell already knows its geometry and asks the images a precise question.

Temporal self-attention is the second pillar and a genuine advantage of the query design. The previous frame's BEV features, warped into the current ego frame using the vehicle's motion, become keys and values that current queries attend to. This gives the model a short memory: it can infer velocity, carry information about objects momentarily occluded, and stabilize the map across frames, all things a single-frame LSS pass cannot do. The temporal recurrence is why BEVFormer and its successors excel at the velocity and attribute metrics of the nuScenes benchmark, not just position.

A minimal splat: the geometric core in code

The distinctive engineering of LSS is the geometry, not the network. The snippet below builds the frustum of 3D points for one camera and shows how each pixel-depth pair maps to a world coordinate ready for the BEV scatter. Seeing the coordinate transform explicitly demystifies what both paradigms ultimately rely on.

import torch

def frustum_to_ego(depths, H, W, K, cam_to_ego):
    """Map every (pixel u, v, depth d) to a 3D point in the ego frame.
    depths: (D,) depth-bin centers. K: (3,3) intrinsics.
    cam_to_ego: (4,4) camera-to-ego extrinsic. Returns (D,H,W,3)."""
    us = torch.arange(W).view(1, 1, W).expand(len(depths), H, W)
    vs = torch.arange(H).view(1, H, 1).expand(len(depths), H, W)
    ds = depths.view(-1, 1, 1).expand(len(depths), H, W)
    # back-project pixels to camera-frame rays, scaled by depth d
    ones = torch.ones_like(us)
    pix = torch.stack([us * ds, vs * ds, ds], dim=-1).float()   # (D,H,W,3)
    cam_pts = pix @ torch.inverse(K).T                          # undo intrinsics
    hom = torch.cat([cam_pts, ones.unsqueeze(-1).float()], dim=-1)
    ego = hom @ cam_to_ego.T                                    # to shared frame
    return ego[..., :3]                                         # drop homog. w

The function above is the exact transform every BEV cell in BEVFormer also computes, run in reverse: BEVFormer projects known 3D reference points into the image with \(K\) and the extrinsic, where LSS projects image pixels out to 3D. Same geometry, opposite direction of travel.

Choosing between push and pull

The decision reduces to a few honest tradeoffs. LSS is conceptually simple, embarrassingly parallel, and fast, but its accuracy is hostage to the predicted depth distribution, and depth supervision (from lidar, when available) is close to mandatory for competitive results. BEVFormer needs no explicit depth, spends its capacity on attention rather than a depth head, and gains a temporal memory almost for free, at the cost of a heavier deformable-attention kernel and more careful engineering. In practice, camera-only detectors trending toward the top of public benchmarks blend the two: a depth-supervised forward projection to guarantee every cell has support, plus query-based backward attention and temporal fusion to refine and remember. Both feed the same downstream consumer, the shared BEV feature map, which is precisely why the next section can fuse a camera BEV stream with a lidar BEV stream without either encoder knowing the other exists.

What's Next

In Section 43.3, we cash in the shared bird's-eye-view representation that both paradigms produce. Because a camera BEV stream and a lidar BEV stream now live on the same metric grid, fusing them (the BEVFusion recipe) becomes a matter of concatenating feature maps rather than wrestling with cross-modal geometry, and we will see why that simple move is so robust when one sensor degrades or drops out entirely.