Point-cloud representations; voxelization and pillars

"A point cloud is a bag of numbers that refuses to sit still on a grid. Half of lidar research is politely convincing it to."

A Tidying-Up AI Agent

This section builds directly on the raw point-cloud data structure from Section 42.1 and on the general idea, introduced in Chapter 40, that 3D measurements can live either on an image-like grid or as free-floating points. We assume the neural-representation vocabulary of Chapter 13: embeddings, pooling, and why a network must respect the symmetries of its input. Our job here is narrow and load-bearing: define the encodings, derive the pillar pipeline that the sparse-convolution backbones of Section 42.3 and the transformers of Section 42.4 consume, and name the tradeoffs that decide which one you ship.

Why raw points resist standard networks

A point cloud \(P = \{\mathbf{p}_i\}_{i=1}^{N}\) has three properties that break naive convolution. It is unordered: any function that consumes it must be invariant to permutations of the \(N\) points, so a network cannot simply flatten the list into a vector. It is variable in size: \(N\) changes every frame, from a dense near-field return to a sparse distant one. And it is irregularly sampled: density falls off roughly as \(1/r^2\) with range and clusters along scan lines, so there is no fixed neighbor stencil. PointNet-style architectures address permutation invariance directly by applying a shared multilayer perceptron to each point and then pooling with a symmetric operator such as max, but pure point methods struggle to scale to the full-scene, hundred-thousand-point clouds of automotive lidar. The dominant alternative is to discretize: impose a grid, and let each cell aggregate the points that fall inside it. That single move restores order, fixes size, and regularizes density, at the cost of quantization.

Voxelization: a 3D grid of learned cell features

Voxelization partitions space into a regular 3D lattice of cubic cells (voxels) of side \(v_x \times v_y \times v_z\) over a defined range, then assigns each point to the voxel containing it. The voxel index of a point is simply

The influential VoxelNet architecture (Zhou and Tuzel, 2018) turned each non-empty voxel into a learned feature vector with a Voxel Feature Encoding (VFE) layer: a small PointNet applied to the points inside that voxel, producing one fixed-length embedding per occupied cell. The result is a sparse 4D tensor (three spatial axes plus channels) that a 3D convolutional network can process. The decisive practical fact is sparsity: in a typical outdoor scene, well over 90 percent of voxels are empty, because lidar only sees surfaces, not filled volume. Running dense 3D convolutions over that mostly-empty grid is ruinously expensive, which is exactly the problem that sparse convolution, the subject of Section 42.3, was invented to solve. For now, hold onto the shape of the output: a set of occupied cells, each carrying a coordinate and a feature vector.

Pillars: collapse the vertical axis and go 2D

PointPillars (Lang et al., 2019) made a shrewd simplification. Instead of dividing height into many voxels, it uses a single voxel that spans the full vertical extent: a pillar, an infinitely tall column with a small \(x\)-\(y\) footprint. Every point is binned into one pillar by its horizontal position alone. A PointNet-style encoder turns the points in each pillar into a feature vector, and those vectors are scattered back to their \((x, y)\) locations to form a dense 2D pseudo-image: a bird's-eye-view feature map with channels where a camera image would have RGB. The payoff is enormous. A 2D backbone, the same kind of efficient convolutional network the rest of computer vision has spent a decade optimizing, now does the heavy lifting, and there is no expensive 3D convolution anywhere in the pipeline. The bird's-eye-view output feeds naturally into the shared BEV representation developed in Chapter 43.

The cost of pillars is vertical resolution: by collapsing the \(z\) axis into learned channels rather than explicit cells, the network must infer height structure from the pooled features instead of reading it off a grid. For road scenes, where objects sit on a roughly flat ground plane and their footprint is highly discriminative, this loss barely hurts. For scenes with rich vertical stacking, or for tasks needing fine height detail, keeping explicit voxels pays off. That is the whole voxel-versus-pillar decision in one sentence: pillars win on speed and simplicity; voxels win when vertical geometry carries signal.

import numpy as np

def scatter_to_pillars(points, x_range=(0, 70), y_range=(-40, 40),
                       cell=0.16, max_points=32):
    """Bin an (N,4) xyz+intensity cloud into a BEV pillar grid.
    Returns a dense (C, H, W) pseudo-image of simple hand-crafted
    per-pillar features (a real net replaces these with a PointNet)."""
    xs, ys = points[:, 0], points[:, 1]
    keep = ((xs >= x_range[0]) & (xs < x_range[1]) &
            (ys >= y_range[0]) & (ys < y_range[1]))
    pts = points[keep]
    gx = ((pts[:, 0] - x_range[0]) / cell).astype(np.int64)
    gy = ((pts[:, 1] - y_range[0]) / cell).astype(np.int64)
    W = int((x_range[1] - x_range[0]) / cell)
    H = int((y_range[1] - y_range[0]) / cell)
    flat = gy * W + gx                       # one id per pillar
    C = 4                                     # count, mean-z, max-z, mean-intensity
    img = np.zeros((C, H * W), dtype=np.float32)
    order = np.argsort(flat)
    flat, pts = flat[order], pts[order]
    ids, start = np.unique(flat, return_index=True)
    for k, s in zip(ids, start):
        e = start[np.searchsorted(start, s) + 1] if s != start[-1] else len(flat)
        col = pts[s:e][:max_points]           # cap points per pillar
        img[0, k] = len(col)
        img[1, k] = col[:, 2].mean()
        img[2, k] = col[:, 2].max()
        img[3, k] = col[:, 3].mean()
    return img.reshape(C, H, W)

The code above strips PointPillars to its skeleton so the moving parts are explicit. Real implementations replace the four hand-crafted channels with a learned encoder and augment each point with its offset from the pillar center and from the pillar's mean, which restores some of the sub-cell position the binning discarded. But the control flow, bin horizontally, cap per pillar, scatter to a dense BEV image, is exactly what ships.

Range images and choosing among the three

A fourth encoding deserves mention: the range image, which exploits the fact that a spinning lidar naturally produces an organized grid indexed by beam elevation and azimuth. Unprojecting is unnecessary; the sensor already hands you a dense 2D array whose pixels are ranges, so an ordinary 2D convolutional network applies directly and runs very fast. The catch is that objects change apparent size and shape with distance in a range image, and neighboring pixels can be far apart in 3D, which hurts the metric reasoning that bird's-eye-view representations make easy. In practice: reach for pillars as the default for real-time BEV detection on a compute budget; escalate to voxels with sparse convolution when vertical structure or fine localization matters and you can afford the 3D backbone; consider range images when raw latency dominates and your task tolerates their view-dependent geometry; and keep point-native methods (Chapters 42.4) for indoor or object-scale clouds where \(N\) is modest and every point counts.

What's Next

In Section 42.3, we confront the sparsity we kept flagging head-on. Sparse-convolution backbones such as CenterPoint and VoxelNeXt compute only on occupied cells, skipping the empty 90 percent entirely, which is what makes voxel-based detection fast enough to matter. We will see how the occupied-cell tensors defined in this section become the input to those backbones and how a fully sparse pipeline predicts objects without ever paying for the empty air between them.