Benchmarks and metrics

"Give me a metric that rewards thick walls and I will paint you the thickest, most confident, most wrong wall you have ever scored ninety percent on."

An Adversarially Honest AI Agent

This section is the accounting department for the whole chapter. It assumes you know what a BEV feature map and a 3D occupancy grid are (Sections 43.1 through 43.4), and it leans hard on two foundations from elsewhere in the book: the leakage-safe splitting discipline of Chapter 5, and the calibration and uncertainty view of Chapter 18, because a confident occupancy prediction that is systematically overconfident games most volumetric metrics. Our narrow job here: name the datasets, define the metrics precisely enough to compute them, and expose the failure modes that make a good-looking score untrustworthy.

Three tasks, three metric families

BEV perception is not one benchmark. It splits into three tasks that are scored in genuinely different ways, and confusing them is the most common way to misread a paper's table.

3D object detection asks for oriented boxes with a class, position, size, heading, and velocity. It is scored on nuScenes (Caesar et al., 2020), Waymo Open (Sun et al., 2020), and Argoverse 2. BEV map segmentation asks for a rasterized top-down semantic map (drivable area, lane divider, pedestrian crossing) and is scored with per-class intersection-over-union. 3D semantic occupancy, the task of Sections 43.4 and 43.5, asks for a dense voxel grid where every cell is empty or carries a semantic label, and is scored on Occ3D-nuScenes, Occ3D-Waymo (Tian et al., 2023), SurroundOcc (Wei et al., 2023), and the older semantic scene completion track of SemanticKITTI (Behley et al., 2019). Each family has a signature metric, and the rest of this section takes them one at a time.

Detection: why nuScenes replaced plain mAP with NDS

Classical detection uses mean average precision, where a prediction counts as a true positive if its box overlaps a ground-truth box above an IoU threshold. In 3D that overlap is brittle: a box off by half a meter in a busy scene can flip from hit to miss on a threshold you did not choose for a good reason. The nuScenes benchmark made two decisions that the field has largely adopted. First, it matches predictions to ground truth by 2D center distance on the ground plane (thresholds of 0.5, 1, 2, and 4 meters) rather than by IoU, decoupling the match decision from box shape. Mean average precision is then averaged over those four distance thresholds and over classes.

Second, and more important, nuScenes recognized that a detector can hit the right center yet be wrong about everything else, so it added five true-positive error metrics computed over matched boxes: translation (mATE), scale (mASE), orientation (mAOE), velocity (mAVE), and attribute (mAAE) error. These combine with mAP into the nuScenes Detection Score,

where \(\mathbb{TP}\) is the set of five error metrics. The weighting is deliberate: mAP carries half the score, and the five geometric and dynamic errors share the other half. A model that finds objects but misjudges their heading or velocity is penalized in a way plain mAP never captured, which matters because a self-driving planner acts on velocity and heading, not on the fact that something is roughly there.

Occupancy: voxel mIoU and the thickness it silently rewards

The default occupancy metric is mean intersection-over-union over voxels. For each semantic class \(c\), count voxels that are correctly predicted as \(c\) (true positives \(TP_c\)), predicted \(c\) but actually not (false positives \(FP_c\)), and actually \(c\) but predicted otherwise (false negatives \(FN_c\)), then

SemanticKITTI's semantic scene completion track reports this alongside a class-agnostic completion IoU that scores only occupied-versus-empty geometry, so you can see whether errors are about where the surface is or what it is. Voxel mIoU is intuitive and it is what Occ3D and SurroundOcc leaderboards rank on. It also has a quietly serious flaw, discovered when researchers looked closely at what high-scoring models actually predict.

Occupancy ground truth is built by accumulating lidar sweeps, so surfaces are labeled a few voxels deep. A model can inflate its IoU by predicting surfaces thicker than they are: extra occupied voxels behind a true wall overlap nothing visible from the sensor, cost little in false positives relative to the true-positive mass they add, and are never penalized because the ground truth behind the wall is unobserved. The result is a leaderboard that rewards puffy, over-confident geometry that would wreck a planner trying to thread a gap. The RayIoU metric of SparseOcc (Liu et al., 2024) fixes this by scoring along sensor rays: it casts query rays into the predicted volume, finds the first occupied voxel each ray hits, and compares that depth-and-label to the ground truth along the same ray. A wall that is too thick no longer helps, because only the first hit counts, and depth error is measured directly. This is the same "score what the sensor could actually observe" principle that runs through Chapter 65.

import numpy as np

def occupancy_miou(pred, gt, num_classes, visible_mask=None, empty_id=0):
    """Semantic mIoU over a voxel grid, restricted to observed voxels.

    pred, gt: int arrays of the same shape, class id per voxel.
    visible_mask: bool array, True where the voxel is camera-observed.
    empty_id is excluded so 'guess empty everywhere' cannot win.
    """
    if visible_mask is not None:            # do not score behind occluders
        pred, gt = pred[visible_mask], gt[visible_mask]
    ious = []
    for c in range(num_classes):
        if c == empty_id:
            continue
        p, g = (pred == c), (gt == c)
        inter = np.logical_and(p, g).sum()
        union = np.logical_or(p, g).sum()
        if union == 0:                      # class absent from this scene
            continue
        ious.append(inter / union)
    return float(np.mean(ious)) if ious else 0.0

The code above shows why the camera-visibility mask that Occ3D ships is not a detail. Without it, a model is graded on voxels behind walls and outside any sensor's view, which no method can predict from evidence, so the metric measures dataset priors instead of perception. Occ3D reports mIoU both with and without the mask; always check which one a table uses before comparing across papers.

Evaluating for the real world: leakage, robustness, and missing sensors

A metric is only as trustworthy as the split it runs on. nuScenes, Waymo, and Argoverse partition by scene and location so that consecutive frames from one drive cannot straddle train and test; without that, temporally adjacent frames leak and every score inflates. This is the geographic and session-level splitting rule of Chapter 5 applied to driving logs, and it is the first thing to verify before trusting any custom occupancy benchmark you build in-house.

Second, an averaged score hides where a model breaks. Report per-class mIoU (occupancy leaderboards are dominated by "drivable surface" and "car"; rare classes like "bicycle" or "traffic cone" are where fusion earns its keep) and slice detection results by range, night versus day, and rain. Third, and central to this chapter's lab, evaluate under missing-sensor conditions. A fused BEV model should be graded not only in the all-sensors-healthy case but with a camera blinded or the lidar dropped, because that is a real failure mode on a real vehicle. Robustness curves of this kind belong to the distribution-shift toolkit of Chapter 66, and they routinely reorder a leaderboard whose top entry was quietly leaning on one modality.

What's Next

In Chapter 44, we leave the camera-and-lidar world for radar, a sensor that sees through fog and rain and measures velocity directly but paints the scene in sparse, noisy, low-resolution returns. We will build learning systems on radar tensors and point clouds, and you will see the benchmark discipline of this section reappear immediately, because radar's failure cases and its natural metrics differ enough from lidar's that borrowing nuScenes numbers uncritically is its own kind of leakage.