Evaluation and benchmarks

"A trajectory that looks perfect is one you forgot to align to ground truth."

A Skeptical AI Agent

Trajectory accuracy: ATE, RPE, and the alignment you must not skip

The first question any SLAM evaluation answers is how close is the estimated path to the truth, and there are two construct-distinct ways to ask it. Absolute Trajectory Error (ATE) measures global consistency: it takes the whole estimated trajectory, rigidly aligns it to the ground-truth trajectory, and reports the root-mean-square of the per-pose position differences. Relative Pose Error (RPE) measures local drift: it compares the estimated and true motion over fixed time or distance intervals \(\Delta\), so a system that tracks smoothly but slowly accumulates yaw error scores well on RPE and badly on ATE. You report both because they answer different questions. ATE tells the AR designer whether the lamp stays pinned to the desk; RPE (often stated as drift in percent of distance travelled, the KITTI convention) tells the robotics engineer how fast error grows per metre.

The alignment step is where most published comparisons quietly break. A stereo or lidar system produces a metric trajectory and should be aligned with a rigid \(SE(3)\) transform (rotation and translation only). A monocular system has an unobservable scale, exactly the metric-scale gap discussed for dead reckoning in Chapter 24, so it must be aligned with a similarity transform \(Sim(3)\) that also fits one global scale factor. The closed-form solution is Umeyama alignment. Given estimated positions \(\{\mathbf{p}_i\}\) and truth \(\{\mathbf{q}_i\}\), ATE is

where you solve for \(s=1\) in the metric case and for a free \(s\) in the monocular case. Aligning a monocular result with a rigid transform makes it look ten times worse than it is; aligning a stereo result with a free scale silently hides real scale drift. The single most common leaderboard error is a scale mismatch between the two.

import numpy as np

def umeyama_ate(P, Q, fix_scale=False):
    """ATE with SE(3) (fix_scale=True) or Sim(3) (monocular) alignment.
    P: estimated Nx3 positions, Q: ground-truth Nx3 positions (time-matched)."""
    muP, muQ = P.mean(0), Q.mean(0)
    Pc, Qc = P - muP, Q - muQ
    Sigma = (Qc.T @ Pc) / len(P)
    U, D, Vt = np.linalg.svd(Sigma)
    S = np.eye(3)
    if np.linalg.det(U) * np.linalg.det(Vt) < 0:   # reflection guard
        S[2, 2] = -1
    R = U @ S @ Vt
    s = 1.0 if fix_scale else np.trace(np.diag(D) @ S) / (Pc**2).sum() * len(P)
    t = muQ - s * R @ muP
    err = Q - (s * (P @ R.T) + t)                    # per-pose residuals
    return np.sqrt((err**2).sum(1).mean()), s        # ATE (RMSE), fitted scale

Listing 52.7 makes the alignment explicit so the choice cannot hide in a library call. In practice you run it after time-synchronizing the two trajectories, because estimate and ground truth arrive on different clocks (the synchronization problem of Chapter 3); a 20 ms timestamp offset during fast motion inflates ATE more than most algorithmic differences you are trying to measure.

Map and rendering quality: geometry versus appearance

Trajectory error says nothing about the map, and the neural and Gaussian-splatting systems of Sections 52.3 through 52.5 exist precisely to produce a good map. Map quality splits into two construct-distinct families. Geometric reconstruction is scored against a ground-truth mesh or depth: accuracy (mean distance from each reconstructed point to the nearest true surface), completeness (the reverse distance, penalizing holes), depth L1 error, and the combined F-score at a distance threshold, which is the geometric analogue of precision and recall. A system can be accurate but incomplete (sharp where it commits, full of holes elsewhere) or complete but inaccurate (dense but smeared), so you report both halves, never just one.

Appearance / novel-view quality matters for the radiance-field and splatting systems built on the representations of Chapter 51, where the map is judged by how it looks from held-out camera poses. The standard triple is PSNR (pixel fidelity, easily gamed by blur), SSIM (structural similarity), and LPIPS (a learned perceptual distance that correlates far better with human judgement). Report all three: a splatting map can win PSNR by over-smoothing while losing LPIPS, which is the metric a human actually notices. The critical protocol point is that novel views must be genuinely held out from the mapping stream, or you are measuring memorization, not reconstruction, the exact leakage failure mode formalized in Chapter 65.

Datasets, benchmarks, and what each one actually tests

No single dataset stresses everything, so competent evaluation reports a suite. EuRoC MAV and TUM-VI are the visual-inertial standards, with Vicon or motion-capture ground truth and controlled indoor motion; they test drone- and headset-style dynamics. KITTI odometry is the outdoor driving benchmark, long metric trajectories where per-distance drift dominates. TUM RGB-D and ICL-NUIM are the classic indoor RGB-D sets, ICL-NUIM notable for a synthetic ground-truth mesh that makes reconstruction accuracy and completeness computable exactly. Replica and ScanNet drive most neural and splatting SLAM evaluation because they pair posed RGB-D with dense mesh and semantics. TartanAir adds hard photometric and dynamic conditions, and the Newer College and Hilti SLAM Challenge sets push lidar-inertial-camera fusion in construction sites where GNSS is unavailable. Choose datasets whose motion profile and sensor suite match your deployment; an ATE that is excellent on smooth EuRoC flights tells a car nothing.

Protocol: making a comparison you can defend

A defensible SLAM comparison rests on four rules. First, hold the input identical: same sequences, same sensor calibration, same start-up excitation, or you are comparing datasets, not systems. Second, report variance: neural and splatting SLAM are nondeterministic, so a single run is an anecdote; report median and spread over at least five seeds, because a system with a great mean and catastrophic tail is unshippable. Third, measure the cost, not just the accuracy: real-time factor (does it keep up with the sensor stream?), peak memory, and end-to-end latency belong in the same table as ATE, since a system that needs 4x real time is disqualified for a live headset regardless of its error. Fourth, test the failure modes on purpose: inject camera dropout, dynamic objects, and low texture, and report degradation, not just the clean number, because the clean number is the one condition a deployed system rarely enjoys. These four rules are the SLAM-specific instance of the general leakage-safe evaluation discipline that runs as a thread through this whole book.

What's Next

In Chapter 53, we move from mapping the world as it is to modelling how it will change: world models that predict future sensor observations and let an agent plan in a learned latent space. Evaluation shifts with the goal, from trajectory error against a static ground truth to physical-plausibility and prediction benchmarks, and the leakage-safe discipline built here carries directly into judging whether a model has learned dynamics or merely memorized a rollout.