"My trajectory scored a beautiful 4 centimeters of error. I had quietly rotated and stretched the estimate until it snapped onto the truth. The reviewer asked what the number meant. I no longer knew."
An Overfitted AI Agent
Why this section matters
Every method in this chapter, from the complementary filter of Section 24.2 to the learned inertial odometry of Section 24.6, produces a stream of estimated poses. The question that decides which one you ship is deceptively simple: how wrong is the estimated path? It turns out there is no single answer, because a trajectory can be wrong in two very different ways. It can be globally warped, sitting in the wrong place and pointing the wrong way, or it can be locally jittery while drifting slowly over distance. These call for different metrics: absolute trajectory error and relative pose error. Getting the comparison right also demands a subtle alignment step, honest handling of orientation on a curved manifold, and enough statistical discipline that the number you report survives being run on data the model never saw. This section is the ruler the rest of the chapter is measured with, and the ruler you will carry into localization, SLAM, and odometry benchmarking for the rest of the book.
We assume you can produce a time-stamped sequence of estimated poses and have a reference trajectory to compare against. Reference truth usually comes from motion capture (Vicon or OptiTrack, sub-millimeter but confined to a room), GNSS-RTK (centimeter-level outdoors), or a surveyed total station. Aligning two clocks before comparing anything is non-negotiable; if timestamps are off, revisit synchronization in Chapter 3. Because we compare rotations, the geodesic distance on \(SO(3)\) from Section 24.1 reappears here as an error metric. And the leakage-safe evaluation discipline that governs every trained model in this book (Chapter 65) applies with full force the moment any part of the pipeline is learned.
Two ways a trajectory is wrong: global versus local error
Picture an estimated indoor walk that is a near-perfect copy of the true path but rotated fifteen degrees about the start and shifted half a meter. Point-by-point, the position error is large and grows with distance from the origin, yet the shape is flawless. Now picture a second estimate that hugs the true path closely everywhere but shivers with high-frequency noise. The first has a global consistency error; the second has a local one. Reporting only one number hides the other. The field therefore standardized on two complementary metrics, popularized by the TUM RGB-D benchmark. Absolute Trajectory Error (ATE) measures global consistency: how far the whole estimated path sits from truth after best-fit alignment. Relative Pose Error (RPE) measures local accuracy and drift: how much the pose change over a short window disagrees with truth, independent of any global offset. A method can win on one and lose on the other, which is precisely why you report both.
Absolute Trajectory Error and the alignment trap
ATE cannot be computed on raw coordinates, because the estimate and the reference may live in different frames: an inertial navigator starts at an arbitrary origin and heading. So we first align. Find the rigid transform (rotation \(\mathbf{R}\), translation \(\mathbf{t}\), and for scale-ambiguous methods a scale \(s\)) that best maps the estimated positions onto the reference in a least-squares sense. The closed-form solution is the Umeyama algorithm, a singular-value decomposition of the cross-covariance of the two point sets, the trajectory cousin of Horn's absolute-orientation solution. After alignment the per-sample error and its root-mean-square are
$$ \mathbf{e}_i = \mathbf{p}^{\text{gt}}_i - \big(s\mathbf{R}\,\hat{\mathbf{p}}_i + \mathbf{t}\big), \qquad \text{ATE}_{\text{RMSE}} = \sqrt{\tfrac{1}{N}\textstyle\sum_{i=1}^{N} \lVert \mathbf{e}_i \rVert^2}. $$The alignment is also the trap. A pure inertial dead-reckoning system has no absolute scale or heading reference, so aligning with a full similarity transform (\(Sim(3)\), scale free) is fair; but running that same generous alignment on a method that does claim metric scale silently forgives its scale error and flatters the number. The rule: align with the weakest transform the method genuinely leaves unobservable, and no weaker. For visual-inertial odometry that is usually a 4-degree-of-freedom yaw-plus-translation alignment, because roll, pitch, and scale are observable from gravity and inertial cues, and pretending otherwise is a way to launder error.
import numpy as np
def umeyama_align(src, dst, with_scale=True):
# src, dst: (N,3) estimated and reference positions, time-synced.
mu_s, mu_d = src.mean(0), dst.mean(0)
S, D = src - mu_s, dst - mu_d
cov = (D.T @ S) / len(src)
U, d, Vt = np.linalg.svd(cov)
W = np.eye(3)
if np.linalg.det(U) * np.linalg.det(Vt) < 0: # reflection guard
W[2, 2] = -1
R = U @ W @ Vt
s = (d * np.diag(W)).sum() / (S ** 2).sum() * len(src) if with_scale else 1.0
t = mu_d - s * R @ mu_s
return R, t, s
def ate_rmse(est, gt, with_scale=False):
R, t, s = umeyama_align(est, gt, with_scale)
aligned = (s * (R @ est.T).T) + t
err = np.linalg.norm(gt - aligned, axis=1)
return np.sqrt(np.mean(err ** 2)), err
umeyama_align returns the best-fit rotation, translation, and (optionally) scale via one SVD; ate_rmse applies it and reports the RMSE plus the per-sample error vector for plotting. Set with_scale=True only for methods that genuinely lack metric scale, such as pure monocular or scale-blind inertial odometry.RPE isolates drift because it never aligns globally
Relative Pose Error compares pose increments over a fixed span \(\Delta\) (a time interval or a travelled distance). For each \(i\) it forms the relative motion \(\hat{\mathbf{T}}_i^{-1}\hat{\mathbf{T}}_{i+\Delta}\) from the estimate and the same from truth, then measures how they differ. Because it only ever looks at local increments, a constant global offset or heading error cancels out entirely: RPE sees drift, not placement. This is exactly why the KITTI odometry benchmark reports translational and rotational error averaged over sub-sequences of 100 to 800 meters, quoting a percent-of-distance drift and a degrees-per-meter rotation rate. For dead reckoning that percent-of-distance figure, roughly 0.2 to 1 percent for a good foot-mounted ZUPT navigator (Section 24.5), is the honest headline number, because it is invariant to where you pinned the origin.
Orientation error lives on a curved manifold
Position error is a Euclidean distance and behaves. Orientation error does not: rotations form the curved group \(SO(3)\), so subtracting Euler angles is wrong (it ignores wraparound and gimbal coupling) and averaging quaternions naively is wrong too. The correct scalar error between an estimated rotation \(\hat{\mathbf{R}}\) and truth \(\mathbf{R}\) is the geodesic angle, the magnitude of the smallest rotation carrying one onto the other:
$$ \theta = \arccos\!\Big(\frac{\operatorname{tr}(\mathbf{R}^\top \hat{\mathbf{R}}) - 1}{2}\Big) = \big\lVert \log(\mathbf{R}^\top \hat{\mathbf{R}}) \big\rVert. $$Report this in degrees, and split heading (yaw) from tilt (roll and pitch) whenever gravity is observable, because they have utterly different error physics: tilt is bounded by an accelerometer that always sees gravity, while yaw drifts without bound absent a magnetic or map reference. An attitude-and-heading system (Section 24.3) that looks excellent on total angular error can still be quietly hemorrhaging yaw, and only the split reveals it. When the estimator also emits a covariance, check that the error is calibrated against it, the same calibration and reliability question raised for every uncertainty estimate in Chapter 18.
Reporting numbers you can defend
A single RMSE on a single trajectory is a coin flip dressed as a result. RMSE is dominated by its worst moments, so pair it with the median and a percentile (the CDF of position error tells the real story), and prefer the median for skewed error distributions. Never pool samples across trajectories into one giant RMSE: a long trajectory then drowns out short ones and one lucky run hides a systematic failure. Report per-trajectory statistics and aggregate across trajectories, ideally with confidence intervals from multiple runs or seeds, since learned inertial models vary run to run. Most important, the moment any component is trained, the reference must be held out by subject and by session, not merely by time window, or motion signatures leak between train and test and the pretty ATE is measuring memorization. This leakage-safe split is the through-line of Chapter 65, and it is where most inertial-odometry papers that later fail to reproduce went wrong.
Frontier: metrics that keep up with learned odometry
Classic ATE and RPE were designed for geometric SLAM. Learned inertial navigators such as RoNIN and the IONet line (Section 24.6) exposed their gaps. RoNIN popularized the Relative Trajectory Error over fixed time windows (for example one minute) as a drift measure decoupled from path length, plus Position Drift and a heading-consistency term, because velocity-regressing networks fail differently from filters. The current frontier evaluates the full predictive distribution, not just the mean path: does the reported covariance actually cover the true pose, and does it stay calibrated under distribution shift onto new users, gaits, and mounting positions (Chapter 66)? Zhang and Scaramuzza's quantitative-evaluation tutorial and the widely used evo toolkit are the community's converging answer for reproducible, alignment-honest reporting.
Grading two pedometers for a stroke-rehab wearable
A clinical team evaluates two wrist trackers that estimate a patient's walking path during a supervised gait test, one a hand-tuned complementary-filter-plus-step-model pipeline, the other a learned inertial-odometry network. On the demo clip both look identical when overlaid on the OptiTrack reference. The evaluation tells a sharper story. Aligned with the correct yaw-only transform, the classical pipeline posts a lower ATE on straight walks but its RPE balloons on turns, where the step-heading model mis-attributes arm swing to heading. The learned model has steadier RPE across turns but its per-trajectory ATE variance is triple the classical one, and crucially its error jumps on the two left-hemiparetic patients whose asymmetric gait never appeared in training. Because the team split the reference by patient, that generalization gap showed up as an honest metric rather than a deployment surprise. They ship the classical pipeline for the current cohort and flag the learned model for retraining on impaired gait.
Right tool: evo for alignment-honest trajectory evaluation
The alignment, ATE, RPE, per-segment KITTI drift, plotting, and multi-trajectory statistics above add up to a few hundred lines you would otherwise babysit. The open evo package does all of it from the command line or a dozen Python lines: evo_ape and evo_rpe read TUM, KITTI, EuRoC, or ROS-bag formats, apply Umeyama alignment with a chosen degrees-of-freedom flag (--align, --correct_scale, --n_to_align), and emit RMSE, median, and CDF plots that reviewers already recognize. Reach for the from-scratch code once to internalize the alignment trap, then standardize on evo so your numbers are comparable to everyone else's and the alignment choice is explicit in the command that produced them.
Exercise
Take one estimated trajectory and its reference. (1) Compute ATE three ways: no alignment, yaw-plus-translation alignment, and full \(Sim(3)\) alignment with scale. Report all three and explain which is defensible for a metric-scale inertial navigator and why the scaled one is not. (2) Compute RPE over window lengths of 1, 10, and 60 seconds and plot how the drift rate changes with window size. (3) Inject a constant 10-degree yaw offset into the whole estimate and recompute both ATE and RPE. Which metric barely moves, and what does that tell you about what each one measures?
Self-check
1. An estimated path has near-zero RPE but large ATE. Describe in words what the trajectory looks like relative to truth.
2. Why is aligning a metric-scale trajectory with a scale-free \(Sim(3)\) transform a form of cheating, and what does it hide?
3. Why must orientation error use the geodesic angle rather than the Euclidean difference of Euler angles, and why split yaw from tilt?
Lab 24
implement orientation estimation and compare a complementary filter with a learned inertial-odometry model.
Bibliography
Benchmark metrics and protocols
The TUM RGB-D benchmark paper that defined Absolute Trajectory Error and Relative Pose Error as the standard pair; the source of the ATE/RPE vocabulary used everywhere in trajectory evaluation.
Introduced the odometry metric of averaging translational and rotational error over sub-sequences of 100 to 800 meters, the percent-of-distance drift measure that dominates dead-reckoning reporting.
The definitive practical guide to alignment choices (rigid versus similarity, degrees of freedom) and to reporting ATE and RPE without laundering error; required reading before publishing any trajectory number.
Alignment mathematics
The closed-form SVD solution for the best-fit rotation, translation, and scale between two point sets; the algorithm behind every ATE alignment step, including the code in this section.
The quaternion-based absolute-orientation solution that predates and complements Umeyama; foundational for aligning two coordinate frames from corresponding points.
Tools and learned-odometry evaluation
Grupp (2017). evo: Python Package for the Evaluation of Odometry and SLAM. Open-source software.
The community-standard toolkit implementing ATE, RPE, KITTI drift, and alignment-honest reporting across TUM, KITTI, EuRoC, and ROS formats; the recommended right tool for reproducible evaluation.
Herath, Yan, Furukawa (2020). RoNIN: Robust Neural Inertial Navigation in the Wild. IEEE ICRA.
Popularized Relative Trajectory Error over fixed time windows and position-drift metrics for learned inertial odometry, and modeled the subject-held-out split that keeps such evaluations leakage-safe.
The learned-inertial-odometry work from Section 24.6 whose trajectory-error evaluation established how data-driven navigators are compared against classical dead reckoning.
What's Next
In Chapter 25, we leave the self-contained world of inertial dead reckoning and bring in absolute references. GNSS, RF ranging, and map constraints supply the very heading and position anchors that ATE and RPE just showed us dead reckoning cannot recover on its own, and the trajectory metrics of this section become the yardstick for how much each anchor actually helps.