4D imaging radar detection and occupancy (RadarOcc, RCBEVDet)

"Give me a fourth axis and suddenly I know not just where and how fast, but how tall. Grant me a few hundred virtual antennas and I will draw you a scene through the fog that lidar refused to enter. It will be blurry. It will be honest. It will still be there at midnight in a blizzard."

An Elevated AI Agent

From virtual array to a four-dimensional data product

What a 4D imaging radar produces, before any AI touches it, is a four-dimensional intensity tensor indexed by range, azimuth, elevation, and Doppler, \(I[r,\theta,\phi,v]\). The extra elevation axis \(\phi\) is the whole story: a 2D MIMO array with transmit and receive elements distributed in both a horizontal and a vertical layout synthesizes a large 2D virtual aperture, and the same spatial-frequency estimation that gave azimuth in Section 44.1 now also gives elevation. High-end sensors reach hundreds to low thousands of virtual channels, pushing angular resolution toward (though not to) lidar territory. Why this is not simply "lidar that sees through weather" is density and noise: even a rich 4D radar returns orders of magnitude fewer confident points per frame than a lidar, and those points sit in a sea of sidelobes, multipath ghosts, and clutter. How the community consumes this varies along one crucial axis of choice.

RCBEVDet: radar-camera detection in bird's-eye view

What RCBEVDet (Lin et al., CVPR 2024) does is fuse a 4D radar point cloud with surround-view cameras into a shared BEV feature map and predict 3D boxes there. Why the pairing is natural: cameras give dense semantics and fine azimuth but guess badly at absolute depth and velocity, while radar gives true range and instantaneous radial velocity but sparse, low-resolution structure. In BEV, the two are complementary layers of the same top-down grid, exactly the fusion setting of Chapter 43 and, more generally, the missing-modality robustness theme of Chapter 50. How RCBEVDet earns its numbers is two design ideas worth remembering. First, a dedicated radar backbone, RadarBEVNet, runs a dual stream: a point-based branch and a transformer branch that share information, plus an RCS-aware scatter that spreads each detection's energy over the BEV cells its cross-section implies rather than a single pixel, respecting radar's inherent spatial smear. Second, a cross-attention fusion module aligns the radar and camera BEV features before summing them, because calibration and ego-motion leave the two grids slightly misregistered and a naive add blurs both. On nuScenes and on the View-of-Delft (VoD) 4D-radar benchmark, this radar-camera design closes much of the gap to lidar while keeping radar's all-weather resilience; RCBEVDet++ extends the fusion further. The payoff shows most in rain and at night, where the camera-only baseline degrades and radar holds the velocity and range channels steady.

RadarOcc: dense 3D occupancy from radar alone

What RadarOcc (Ling et al., NeurIPS 2024) targets is the harder, denser task of 3D semantic occupancy, labelling every voxel in a volume around the vehicle as free or occupied (and by which class) from 4D radar, without a camera or lidar crutch. Why this is ambitious: occupancy is unforgiving of sparsity, because an empty voxel that should be filled is a hole an planner can drive into, and radar's sparse points leave most of the grid unobserved. RadarOcc's answer is to refuse the point-cloud path entirely and keep the raw 4D tensor, so no faint return is discarded before the network can use it. How it copes with the tensor's size and its native geometry is the interesting part. Rather than voxelize the sphere into a Cartesian grid up front (which wastes resolution and interpolates away structure), RadarOcc processes the data in its native spherical coordinates, prunes the vast empty regions with a sparsity-aware step, and only then transforms the surviving features into the Cartesian occupancy volume for the final head. Doppler is not collapsed away; the velocity signature helps the network distinguish dynamic occupancy from static. Evaluated on the K-Radar benchmark, RadarOcc shows that radar-only occupancy is feasible and, decisively, that it degrades gracefully in adverse weather where lidar-based occupancy loses returns to rain and snow. That robustness, not raw accuracy, is radar's argument for a seat at the occupancy table.

Building a radar BEV occupancy grid, concretely

Before reaching for a full RadarOcc or RCBEVDet model, it helps to feel how a sparse 4D point cloud becomes the BEV/occupancy tensor those networks consume. The code below scatters a set of radar detections, each with a position, a Doppler velocity, and an RCS, into a top-down BEV grid, accumulating occupancy plus per-cell mean velocity and peak RCS. This is the minimal preprocessing that turns radar's handful of points into the image-like input a BEV backbone expects, and it exposes the sparsity problem viscerally: most cells stay empty.

import numpy as np

def radar_points_to_bev(points, x_range=(-25, 25), y_range=(0, 50),
                        cell=0.5):
    """points: (N,5) array of [x, y, z, doppler, rcs] in meters/(m/s)/dBsm.
    Returns a (C,H,W) BEV tensor: occupancy, mean Doppler, peak RCS."""
    W = int((x_range[1] - x_range[0]) / cell)
    H = int((y_range[1] - y_range[0]) / cell)
    occ = np.zeros((H, W), np.float32)
    dop = np.zeros((H, W), np.float32)
    rcs = np.full((H, W), -1e3, np.float32)      # peak-hold, start very low

    xi = ((points[:, 0] - x_range[0]) / cell).astype(int)
    yi = ((points[:, 1] - y_range[0]) / cell).astype(int)
    keep = (xi >= 0) & (xi < W) & (yi >= 0) & (yi < H)
    xi, yi, p = xi[keep], yi[keep], points[keep]

    for u, v, pt in zip(yi, xi, p):              # accumulate per cell
        occ[u, v] += 1.0
        dop[u, v] += pt[3]
        rcs[u, v] = max(rcs[u, v], pt[4])
    mean_dop = np.divide(dop, occ, out=np.zeros_like(dop), where=occ > 0)
    rcs[occ == 0] = 0.0
    return np.stack([np.clip(occ, 0, 1), mean_dop, rcs])  # (3, H, W)

rng = np.random.default_rng(0)
pts = np.column_stack([rng.uniform(-20, 20, 40), rng.uniform(2, 45, 40),
                       rng.uniform(-1, 3, 40), rng.uniform(-15, 15, 40),
                       rng.uniform(-5, 20, 40)])
bev = radar_points_to_bev(pts)
occupied = (bev[0] > 0).sum()
print(f"grid {bev.shape}, occupied cells {occupied} "
      f"({100*occupied/bev[0].size:.2f}% filled)")

That printed fill fraction, well under one percent, is the entire design tension of this section in one number. The point-cloud path you just coded is fast but leaves the grid almost empty; a downstream network must hallucinate structure into the holes, which is exactly why RadarOcc keeps the dense tensor and why RCBEVDet leans on the camera to fill semantics. The same voxelization generalizes to 3D by adding a height axis, giving the occupancy volume of Chapter 43.

What's Next

In Section 44.5, we widen the fusion story from radar-plus-camera to the full radar-camera-lidar stack and make weather robustness the explicit design goal: how a perception system degrades gracefully when fog blinds the camera, rain thins the lidar, and radar becomes the sensor still standing.