"A dense 3D convolution asked me to multiply a billion zeros by weights I already knew. I would rather spend those cycles on the two thousand voxels that actually caught a photon."
A Parsimonious AI Agent
Prerequisites
This section builds directly on the voxelization and pillar encodings of Section 42.2: we assume a lidar sweep has already become a set of occupied voxels, each carrying a learned per-voxel feature vector, indexed by integer grid coordinates. It uses the ordinary convolution and feature-map vocabulary of Chapter 13, and it produces the bird's-eye-view feature maps that Chapter 43 treats as the shared fusion canvas. The efficiency arguments connect to the model-optimization thinking of Chapter 59. You need no new mathematics beyond counting nonzero grid cells.
Why the backbone is where lidar detection is won or lost
A single automotive lidar sweep, voxelized at 0.1 m over a 100 m range, defines a grid with on the order of \(10^{8}\) cells, of which perhaps \(10^{4}\) to \(10^{5}\) are occupied. That is roughly 0.1 percent occupancy. A dense 3D convolution would spend 99.9 percent of its multiply-accumulate operations grinding zeros against weights, producing zeros. The sparse-convolution backbone is the architecture that refuses to pay that tax: it convolves only where there is data, turning an intractable dense 3D network into something that runs in real time on a car. Every modern lidar detector, including the two we dissect here, CenterPoint and VoxelNeXt, is a sparse backbone plus a detection head. Get the backbone right and the rest is comparatively easy. This section explains the one trick that makes it possible, then shows two production designs that spend the saved compute very differently.
Submanifold sparse convolution: convolving only where data lives
What a sparse convolution does is store the feature map as a list of (coordinate, feature) pairs instead of a filled tensor, and compute outputs only at a chosen set of active sites. Why this is not trivial is a subtle failure mode: a naive sparse convolution still writes an output wherever any kernel tap touches an active input, so a single \(3\times3\times3\) layer dilates a lone occupied voxel into 27 active voxels. Stack a few such layers and the "sparse" feature map fills in, the occupancy climbs toward dense, and the speed advantage evaporates. The fix, introduced by Graham and van der Maaten, is the submanifold sparse convolution (SSC): it produces an output at a site only if that exact site was active in the input. Active sites stay put; the sparsity pattern is preserved layer after layer. Regular (dilating) sparse convolutions are then used sparingly and deliberately, only at stride-2 downsampling steps where you actually want to connect disconnected components and grow the receptive field.
How the computation runs is a three-step gather-matmul-scatter. For each of the \(k^3\) kernel offsets, a rulebook precomputes which active input sites map to which active output sites. The engine gathers those input features into a dense matrix, multiplies by the kernel weight for that offset, and scatters the results back, accumulating into the outputs. Cost scales as \(O(N \cdot k^3 \cdot C_{\text{in}} \cdot C_{\text{out}})\) with \(N\) the number of active sites, not the \(O(HWD \cdot k^3 \cdot C_{\text{in}} \cdot C_{\text{out}})\) of the dense grid. Since \(N \ll HWD\), the saving is three to four orders of magnitude. The small numpy simulation below makes the dilation problem concrete: it counts how the active-site set explodes under a regular sparse convolution but stays fixed under the submanifold rule.
import numpy as np
# 200 occupied voxels scattered in a 400^3 grid (~0.0003% occupancy)
rng = np.random.default_rng(0)
active = set(map(tuple, rng.integers(0, 400, size=(200, 3))))
# 3x3x3 kernel offsets
offsets = [(dx, dy, dz) for dx in (-1,0,1) for dy in (-1,0,1) for dz in (-1,0,1)]
def regular_sparse_out(active, offsets):
out = set()
for (x, y, z) in active: # every tap that touches an active input
for (dx, dy, dz) in offsets: # creates an output -> the set dilates
out.add((x+dx, y+dy, z+dz))
return out
def submanifold_out(active, offsets):
return set(active) # output sites == input sites, always
reg = regular_sparse_out(active, offsets)
sub = submanifold_out(active, offsets)
print(f"input active sites: {len(active)}")
print(f"after 1 regular sparse conv:{len(reg)} ({len(reg)/len(active):.1f}x)")
print(f"after 1 submanifold conv: {len(sub)} (1.0x, sparsity preserved)")
# input active sites: 200
# after 1 regular sparse conv: 5158 (25.8x)
# after 1 submanifold conv: 200 (1.0x, sparsity preserved)Submanifold layers keep, downsampling layers connect
The design rule that emerges is worth memorizing because it explains the block structure of every sparse lidar backbone. Interior feature extraction uses submanifold convolutions, which never touch a voxel that was empty, so two objects a few empty voxels apart never leak features into each other and compute stays flat. Receptive-field growth and cross-object context come only from the strided regular sparse convolutions at stage transitions, where deliberate dilation is the point. A backbone is thus a stack of "many SSC, then one strided sparse conv" blocks, typically four stages that take the grid from 0.1 m voxels down to an 8x-downsampled feature volume ready to be flattened into a bird's-eye-view map.
CenterPoint: a sparse backbone with an anchor-free center head
What CenterPoint (Yin, Zhou, and Krahenbuhl, CVPR 2021) contributes is not a new backbone but the head that finally made sparse backbones easy to train for 3D detection. Its backbone is the standard SECOND-style sparse 3D convolutional stack described above; the innovation is discarding anchors. Classical 3D detectors tiled the BEV plane with pre-sized, pre-oriented anchor boxes, an awkward fit for objects that appear at arbitrary heading. Why anchors hurt in 3D is orientation: a rotated car matches a fixed axis-aligned anchor poorly, and enumerating anchors at many angles is expensive. CenterPoint instead represents each object as a single point, the center of its bounding box, and predicts a BEV heatmap whose peaks are object centers, exactly the keypoint-detection idea from 2D CenterNet lifted into the bird's-eye view. How the box is recovered: at each detected peak the network regresses the remaining degrees of freedom, sub-voxel center offset, height, the three box dimensions, orientation (as \(\sin\theta, \cos\theta\)), and, uniquely for lidar, velocity from two stacked sweeps. A lightweight second stage then pools point features at the predicted box faces to refine confidence and location.
When CenterPoint is the right default is almost any bounded-range automotive or robotics detection task. It is anchor-free, so it handles arbitrary heading gracefully; it predicts velocity, so it feeds tracking directly; and its BEV heatmap head is trivial to extend to new classes. It topped nuScenes and Waymo leaderboards for a long stretch and remains the reference baseline that new methods must beat. Its one structural cost is the dense BEV stage: after the sparse backbone, features are scattered into a filled 2D BEV grid and a dense 2D convolutional neck runs the heatmap head. That dense neck scales with detection area, which is exactly the cost VoxelNeXt sets out to remove.
A delivery robot that never learned "car-sized"
A sidewalk delivery robot runs a 32-beam lidar and must detect pedestrians, cyclists, strollers, dogs, and the occasional scooter, objects whose sizes and headings span a wild range. An early anchor-based prototype needed a hand-tuned anchor set per class and still missed dogs (too small for any anchor) and mishandled cyclists cutting across at 45 degrees. Swapping to a CenterPoint head fixed both without touching the anchor config, because there was none: a dog is just a low, small heatmap peak, and a diagonal cyclist is a peak with a regressed \(\theta\). The team added a "scooter" class by labeling data and adding one heatmap channel, no anchor retuning. The velocity head, fed by two stacked 10 Hz sweeps, handed clean range-rate estimates to the tracker of Section 42.5, so a stopped pedestrian and one stepping off a curb were separable before any Kalman step ran.
VoxelNeXt: going fully sparse
What VoxelNeXt (Chen et al., CVPR 2023) changes is to delete the dense BEV neck and the dense head entirely and predict boxes directly from sparse voxel features. There is no scatter-to-dense-grid step, no dense 2D convolution, no anchors, and no separate refinement stage. Why this matters grows with range. A dense BEV head costs memory and compute proportional to the covered ground area, so extending detection from a 50 m radius to the 200 m needed on highways or in the Argoverse 2 benchmark quadruples the dense grid while the number of actual points barely changes. A fully sparse detector pays only for occupied voxels, so its cost is nearly flat as range grows, the decisive advantage for long-range and large-scene perception. How VoxelNeXt reaches boxes without a dense map is two ideas. First, it adds extra strided sparse downsampling stages so that the receptive field is large enough that a single voxel can "see" a whole object; the network learns to route each object's evidence to one representative sparse voxel near its center. Second, prediction is a sparse max over voxel features that plays the role of CenterPoint's heatmap peak: the winning voxels emit the box regression directly. The result matches CenterPoint accuracy while running faster and scaling gracefully to large detection ranges.
Where sparse backbones are heading
The fully sparse trajectory that VoxelNeXt started is now the frontier. FSD and FSDv2 push fully sparse detection further with instance-level point grouping to recover the fine geometry that pure voxel pooling blurs, targeting the very long-range regime where dense BEV is simply infeasible. In parallel, LargeKernel3D and SpConv 2.x show that large-kernel sparse convolutions, once too slow, become practical with better rulebook kernels and tensor-core gather-scatter, narrowing the historical gap between convolutional backbones and the point transformers of Section 42.4. The open question the field is actively contesting: does the future belong to ever-better sparse convolution, to serialized point transformers, or to hybrids that use sparse convolution for cheap local structure and attention for long-range context?
Do not hand-write the rulebook
The gather-matmul-scatter engine, rulebook construction, CUDA hashing, and half-precision kernels behind a sparse backbone are several thousand lines of GPU code. The spconv library exposes the whole thing as drop-in layers: spconv.SubMConv3d and spconv.SparseConv3d behave like nn.Conv3d but operate on a SparseConvTensor built from your voxel coordinates and features. A four-stage CenterPoint-style backbone is about 40 lines of module definition instead of the roughly 3000 lines of the underlying engine, and OpenPCDet and Pointcept ship complete, trained CenterPoint and VoxelNeXt configs you can fine-tune on a dataset subset in a single command. Reserve the from-scratch view above for building intuition, not for production.
Choosing and deploying a sparse backbone
When to pick which comes down to range and hardware. For bounded ranges (a 50 to 75 m automotive box, an indoor robot), CenterPoint is the safe, well-supported default with the richest ecosystem of pretrained weights and the velocity head that downstream tracking wants. For long-range or very large scenes, or when the dense BEV neck is your latency or memory bottleneck, VoxelNeXt and its fully sparse descendants win by keeping cost tied to point count rather than area. A deployment caveat that bites teams: sparse-convolution kernels are not uniformly supported across inference runtimes. Many embedded accelerators and quantization toolchains handle dense convolutions but not the scatter-gather of spconv, so shipping a sparse backbone to an automotive SoC can force a custom kernel or a fallback, a concrete instance of the deployment-constraint reasoning developed in Chapter 59 and revisited for weather and platform limits in Section 42.7. As always, evaluate on a geographically and temporally disjoint split: lidar scenes from the same drive are near-duplicates, and a random voxel-level split leaks context straight from train to test.
Exercise: measure the sparsity you are protecting
Take any voxelized lidar frame (or the toy grid in the code block). (1) Compute the occupancy fraction: active voxels divided by total grid cells. (2) Using the numpy simulation, count active sites after one, two, and three regular sparse convolutions, and confirm the near-exponential growth, then repeat with submanifold layers and confirm it stays flat. (3) Estimate the multiply-accumulate ratio between a dense \(3\times3\times3\) conv over the full grid and a submanifold conv over the active set for one layer with \(C_{\text{in}}=C_{\text{out}}=64\). What does that ratio tell you about why the dense BEV neck, not the 3D backbone, is CenterPoint's cost center?
Self-check
- Why does a regular sparse convolution destroy the sparsity that made it fast, and what exactly does a submanifold sparse convolution change to prevent that?
- CenterPoint is "anchor-free." What problem specific to 3D boxes does dropping anchors solve, and how does the heatmap head represent an object instead?
- VoxelNeXt removes the dense BEV neck. For which detection scenario does that yield the largest speedup, and why is the benefit small for a short-range indoor robot?
What's Next
In Section 42.4, we replace the convolutional inductive bias with attention: point transformers, and Point Transformer v3 in particular, serialize the point cloud along space-filling curves so that self-attention scales to millions of points, trading the fixed local kernels of a sparse backbone for learned, long-range context. We will see where that trade pays off and where a well-tuned sparse convolution still wins.