BEV multi-sensor fusion (BEVFusion)

"Cameras know what things are and lidar knows where they are. My job is to stop them arguing about it in two different coordinate frames."

A Diplomatic AI Agent

This section assumes the two ingredients that feed it. From Section 43.2 you have the camera-to-BEV lift: how multi-view images become a top-down feature map. From Chapter 42 you have the lidar-to-BEV path: pillars and sparse voxels that scatter a point cloud into a pseudo-image. It also leans on the general fusion taxonomy of Chapter 48, which names the early, mid, and late fusion choices this section will argue between. Our narrow job here: show why the BEV grid is the right meeting place, derive the two-stream architecture that makes it work, and confront the engineering bottleneck that once made it too slow to ship.

Why the fusion point matters more than the fusion trick

Fusion strategies sort by the stage at which the modalities meet. Point-level (early) fusion, exemplified by PointPainting (2020), projects each lidar point into the camera image and stamps it with the semantic-segmentation score of the pixel it lands on. It is simple and it helps, but it inherits lidar's sparsity: a scene with 30,000 lidar points keeps only 30,000 fused samples, discarding millions of camera pixels that hit no point. Proposal-level (late) fusion runs each sensor's detector and merges boxes, which preserves both streams but denies the network any shared representation to learn from, and it fails when one detector never proposes the object at all. Both extremes leak information. The insight of BEVFusion (Liu et al., 2023, at MIT, and a concurrent design by Liang et al., 2022) is that the bird's-eye view is a modality-agnostic feature space: it is metric, it is shared, and crucially it is not owned by either sensor, so neither modality has to be resampled onto the other's structure.

The two-stream architecture

BEVFusion is a Y-shaped network. The camera stream encodes each of the surround-view images with a 2D backbone, predicts a per-pixel depth distribution, and lift-splat-scatters the resulting features into the shared BEV grid exactly as Section 43.2 described, producing a camera BEV feature map \(F_{\text{cam}} \in \mathbb{R}^{C \times H \times W}\). The lidar stream voxelizes or pillarizes the sweep and runs a sparse backbone to produce a lidar BEV map \(F_{\text{lid}} \in \mathbb{R}^{C' \times H \times W}\) on the same \(H \times W\) grid. Because both maps are aligned to identical ground coordinates, fusion is a genuinely local operation: concatenate along the channel axis and let a small convolutional encoder learn the cross-modal combination,

where \(\Vert\) is channel concatenation and \(\phi\) is a few convolutional blocks that reconcile the two feature statistics and correct residual spatial misalignment. A shared detection or segmentation head then reads \(F_{\text{fused}}\). Nothing about the head knows or cares that two sensors produced its input, which is precisely why the same trained backbone can drive 3D detection, map segmentation, and, with the decoders of Section 43.4, occupancy prediction.

The BEV-pooling bottleneck, and the fix that made it real time

The elegant symmetry above hides a nasty cost. Lifting camera features into BEV means each image pixel spawns a column of candidate 3D points along its viewing ray (one per depth bin), and every point must be assigned to a BEV cell and summed with all the others that land there. For six cameras at full resolution this is on the order of two million points per frame scattering into the grid, and a naive implementation of that scatter-and-sum dominated the entire network's latency, often taking longer than both backbones combined. This view-transform cost is the reason early camera-BEV methods were considered too slow to fuse with lidar in a real vehicle.

The MIT BEVFusion contribution that mattered most in practice was not architectural but algorithmic: it made the BEV pooling fast. Two observations do the work. First, the mapping from camera pixels to BEV cells is fixed once the camera calibration and depth-bin geometry are known, so the point-to-cell assignment can be precomputed once and cached rather than recomputed every frame. Second, if you sort those points by their target BEV cell, all points destined for a cell form a contiguous interval, and summing each interval with a specialized interval-reduction GPU kernel avoids the atomic-add contention that cripples the naive version. Together these cut the view transform from the dominant cost to a rounding error, roughly a fortyfold speedup, and turned unified BEV fusion from a research curiosity into something that runs above camera frame rate. The lesson generalizes: in geometric deep learning the deployment blocker is frequently a data-movement kernel, not the math, a theme that returns in the edge-optimization discussion of Chapter 59.

import torch
import torch.nn as nn

class BEVFuser(nn.Module):
    """Fuse aligned camera and lidar BEV feature maps on a shared grid."""
    def __init__(self, c_cam, c_lid, c_out):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Conv2d(c_cam + c_lid, c_out, 3, padding=1, bias=False),
            nn.BatchNorm2d(c_out), nn.ReLU(inplace=True),
            nn.Conv2d(c_out, c_out, 3, padding=1, bias=False),
            nn.BatchNorm2d(c_out), nn.ReLU(inplace=True),
        )

    def forward(self, f_cam, f_lid):
        # Both maps are (B, C, H, W) on the SAME BEV grid.
        if f_cam is None:                        # lidar-only fallback
            f_cam = torch.zeros_like(f_lid[:, :self.encoder[0].in_channels
                                            - f_lid.shape[1]])
        if f_lid is None:                        # camera-only fallback
            f_lid = torch.zeros(f_cam.shape[0], self.encoder[0].in_channels
                                - f_cam.shape[1], *f_cam.shape[2:],
                                device=f_cam.device)
        x = torch.cat([f_cam, f_lid], dim=1)     # channel concat
        return self.encoder(x)                    # (B, c_out, H, W)

The code above makes a quiet but important point that the equation could not: the whole fusion stage is a couple of convolutions. All the difficulty is in producing two aligned inputs, which is why this section spends its budget on the meeting place and the pooling kernel rather than on \(\phi\). The zero-fill branches also foreshadow the next idea.

Robustness: fusion that survives a missing sensor

Because each stream writes into the shared grid independently, either can be dropped without breaking the tensor shapes downstream: substitute zeros for the absent map and the fused head still runs. This is not merely convenient, it is a safety property. Lidar can be blinded by heavy spray or a dirty cover; a camera can be washed out by direct sun or caked in mud. A stack that hard-requires both fails closed at the worst moment. BEVFusion degrades gracefully to the surviving modality, and training with modality dropout, randomly zeroing one stream during training, teaches the fused head to lean on whichever sensor remains rather than collapsing when its favorite disappears. This missing-modality robustness is treated as a first-class objective in Chapter 50; here it falls out almost for free from choosing a shared, sensor-agnostic representation. One caution: graceful degradation depends on tight cross-sensor calibration and time synchronization, because a BEV cell only means the same square meter to both sensors if their extrinsics and timestamps agree, the very concerns raised in Chapter 3.

What's Next

In Section 43.4, we stop asking "where are the objects" and start asking "what is occupied." The same fused BEV features we built here become the input to 3D semantic occupancy prediction, where methods such as Occ3D, SurroundOcc, and TPVFormer label every voxel of the scene as free, occupied, or a specific class, capturing the overhanging branches, oddly shaped debris, and unlabeled obstacles that a fixed box vocabulary can never name.