"They told me the system was accurate to five meters. They did not say five meters was the number I would beat only half the time."
A Statistically Literate AI Agent
The big picture
Every positioning system in this chapter, GNSS, Wi-Fi, UWB, fingerprinting, inertial fusion, produces an estimate that is wrong by some amount, some of the time, in some direction. A single headline number like "accurate to 5 meters" hides all three of those quantifiers and is almost always the wrong summary. This section is the measurement discipline for the whole chapter: how to turn a stack of position errors into honest, comparable, decision-relevant metrics. You will learn why the error distribution is the real object of study, what CEP and R95 actually promise, how safety-critical systems replace accuracy with integrity and protection levels, and which system-level metrics (time to first fix, availability, floor-detection rate) decide whether a technically accurate system is actually usable.
This section assumes the probability vocabulary from Chapter 4, particularly percentiles and the idea that an estimate carries a covariance, not just a value. Because a localization benchmark is an evaluation protocol, the leakage-safe splitting rules from Chapter 5 and Chapter 65 apply directly here, and we return to them for fingerprinting. The metrics compare against a ground-truth trajectory, which in practice comes from a survey-grade RTK receiver, a total station, or a motion-capture rig one to two orders of magnitude more accurate than the system under test.
The error is a distribution, not a number
Start with the atom of every metric: the horizontal position error at time \(t\), the Euclidean distance between the estimate \(\hat{p}_t\) and truth \(p_t\) projected onto the ground plane, \(e_t = \lVert \hat{p}_t - p_t \rVert_2\). Collect these over a test run and you have an empirical error distribution. Everything else is a statistic of that distribution, and the reason the field has so many named metrics is that different statistics answer different questions.
The two most common scalars are the root-mean-square error and a percentile. The horizontal RMSE, \(\sqrt{\frac{1}{T}\sum_t e_t^2}\), is a second-moment summary: it is convenient, it is what least-squares estimators minimize, and it is dangerously sensitive to a handful of large multipath outliers, since squaring turns a single 60-meter blunder into a huge contribution. Percentiles are the robust alternative. CEP, the circular error probable, is the median error: the radius of the circle centered on truth that contains 50 percent of fixes. R95 (sometimes 2DRMS or CEP95) is the 95th percentile, the radius containing 95 percent. The gap between CEP and R95 is itself a diagnostic: a well-behaved Gaussian system has R95 near \(2.1 \times\) CEP, while a system dominated by occasional multipath blunders has a long tail that pushes R95 far higher, a signature you would miss if you reported only the median.
Three habits separate a credible localization report from a marketing sheet. First, always name the percentile: "5 meters" is meaningless, "5 meters CEP" and "5 meters R95" describe very different systems. Second, report the vertical error separately from the horizontal. GNSS vertical error is typically 1.5 to 3 times the horizontal because satellites are only ever above the horizon, never below, weakening the vertical geometry; collapsing the two into one 3D number flatters horizontal accuracy and hides the altitude problem. Third, plot the full cumulative distribution function of \(e_t\), not just two points on it. The CDF is the single most informative localization figure ever drawn: it shows the median, the tail, and lets a reader read off the accuracy at whatever percentile their application cares about.
Key insight
Accuracy and integrity are different questions. Accuracy asks "how large is the typical error?"; integrity asks "what is the largest error the system could have without telling me?" A system can be accurate on average and catastrophically unsafe, if once in ten thousand fixes it is off by 200 meters and reports full confidence. For consumer maps, accuracy is enough. For aircraft approaches, autonomous vehicles, and rail signaling, integrity is the metric that matters, and it is computed and reported entirely differently.
Integrity, protection levels, and the Stanford diagram
Safety-critical positioning does not ask how accurate a fix is; it asks whether the fix can be trusted right now. The machinery, standardized in aviation and increasingly borrowed by automotive, rests on four terms. The alert limit (AL) is the largest position error the application can tolerate before the situation becomes hazardous, a fixed requirement set by the use case (for example a horizontal alert limit of a few meters for lane keeping). The protection level (PL) is a statistical bound the receiver computes in real time from its own geometry and error models: a radius guaranteed to contain the true position with a very high probability, so that the probability of the actual error exceeding PL is below a tiny integrity risk target such as \(10^{-7}\) per hour. Crucially, PL is computed without knowing the truth, from the measurement residuals and satellite geometry alone.
The relationship between the true error, the protection level, and the alert limit defines the system's state, and the canonical way to visualize it over a whole test campaign is the Stanford diagram: a 2D histogram with true error on one axis and protection level on the other. Points below the diagonal (error > PL) are integrity failures, the receiver was more confident than it should have been, and these are the events safety cases are built to make astronomically rare. Points where both error and PL exceed the alert limit are correctly flagged unavailability. The diagram at a glance separates the four regimes: nominal operation, system unavailable (safely), misleading information, and hazardously misleading information. No single scalar captures this, which is why integrity is reported as a diagram plus a set of rates rather than one accuracy figure. This is the same conceptual move as calibrated uncertainty in machine learning, where a model must not only predict but also know when it is likely wrong, developed at length in Chapter 18.
System-level metrics: fast, available, and reliable
A system that is accurate but takes ninety seconds to produce a first fix is useless for a bike-share app, and one that is accurate but drops out in every parking garage fails its users where they most need it. Three operational metrics capture this. Time to first fix (TTFF) is the latency from power-on (or from a request) to the first valid position, and it is reported for cold, warm, and hot starts because a receiver with no recent almanac must download satellite data before it can solve. Availability is the fraction of time (or of attempts) the system delivers a fix meeting a stated accuracy or integrity requirement; a positioning system is characterized by an availability at an accuracy threshold, for example "R95 < 10 m available 92 percent of the time in the test city." Update rate and latency matter for anything that fuses position into a control loop, since a 1 Hz fix delayed by half a second is worthless for the tight inertial coupling of Chapter 24. These metrics interact: aggressive outlier rejection improves accuracy but lowers availability by throwing away marginal fixes, so the honest way to compare two systems is to plot accuracy against availability, not to quote either alone.
import numpy as np
def localization_metrics(est_xy, true_xy):
"""Core horizontal-error metrics from paired estimate/truth arrays (meters)."""
err = np.linalg.norm(est_xy - true_xy, axis=1) # per-fix horizontal error
return {
"cep50_m": float(np.percentile(err, 50)), # median error (CEP)
"r95_m": float(np.percentile(err, 95)), # 95th percentile (R95)
"rmse_m": float(np.sqrt(np.mean(err**2))), # outlier-sensitive
"max_m": float(err.max()), # worst blunder
"tail_ratio": float(np.percentile(err, 95) / np.percentile(err, 50)),
}
def availability(est_xy, true_xy, accuracy_m=10.0):
"""Fraction of fixes meeting an accuracy requirement."""
err = np.linalg.norm(est_xy - true_xy, axis=1)
return float(np.mean(err <= accuracy_m))
rng = np.random.default_rng(0)
truth = rng.uniform(0, 100, size=(2000, 2))
est = truth + rng.normal(0, 3.0, size=(2000, 2)) # 3 m Gaussian noise
est[::200] += rng.normal(0, 40, size=(10, 2)) # 0.5% multipath blunders
print(localization_metrics(est, truth))
print("avail@10m:", availability(est, truth, 10.0))
availability reports the accuracy-versus-coverage tradeoff as a single number at a stated threshold.The snippet above makes the CEP-versus-RMSE contrast concrete: with 0.5 percent of fixes corrupted, the median barely budges while RMSE and the tail ratio jump, telling you at a glance that the system has an outlier problem rather than a precision problem. That is a fault you fix with better multipath rejection (Section 25.1), not with a better antenna.
Practical example: qualifying an asset-tracking wearable for a hospital
A clinical logistics team evaluated a Bluetooth-and-inertial badge for locating infusion pumps across a hospital. The vendor's sheet said "2 meter accuracy," and the first pilot looked fine on the median. But the real requirement was room-level: staff needed to know which room a pump was in, and rooms were 4 meters wide with walls that Bluetooth signals leak through. The team stopped reporting Euclidean error and switched to two categorical metrics: room-level accuracy (fraction of fixes assigned to the correct room) and floor-detection rate (fraction assigned to the correct floor). Room accuracy was only 78 percent despite the flattering 2-meter median, because a badge near a doorway would frequently snap to the neighboring room; the R95 and the CDF tail, not the median, predicted this. Reporting the confusion between adjacent rooms, rather than a distance, was what let the team set a defensible go or no-go threshold and hold the vendor to it.
The hospital story generalizes: for indoor and fingerprinting systems, the meaningful metric is often not distance at all but a discrete outcome, room-level or zone-level accuracy, floor-detection rate, or the mean error reported per building as in the long-running indoor-positioning competitions. And because fingerprinting learns a map from labeled surveys, its evaluation is a supervised-learning evaluation with all the leakage traps that implies. If radio scans from the same visit, or the same short time window, land in both the training fingerprints and the test queries, the reported accuracy is optimistic by meters, because the model is partly memorizing transient radio conditions rather than the venue. The fix is the same temporal and spatial blocking used everywhere in this book: split by visit or by day, never by random scan.
Right tool: do not hand-roll the error-CDF and Stanford plot
The metric functions above are a dozen lines and worth writing once to understand them, but the surrounding machinery, aligning an estimated track to a differently-timestamped ground-truth track by interpolation, handling coordinate frames and geodesic distances, and drawing publication-grade CDF and Stanford diagrams, is a few hundred lines you should not re-implement per project. For trajectory-level evaluation, evo (the SLAM/odometry evaluation toolkit) computes absolute and relative pose error with time alignment in a couple of commands; pyproj handles the geodesy so your distances are metric rather than degrees; and scikit-learn's classification_report gives room-level and floor-level accuracy, precision, and recall for the categorical case for free. Roughly ten lines of client code replaces a few hundred lines of alignment, projection, and plotting logic.
Exercise
Take the code above and extend it. (1) Add a function that plots the empirical CDF of the horizontal error and marks CEP50 and R95 on it; run it on two synthetic systems, one pure-Gaussian and one with a 2 percent heavy tail tuned so both have the same CEP50, and show how the CDFs and R95 values diverge. (2) Sweep the accuracy threshold in availability from 1 to 30 meters and plot availability against threshold: this availability-accuracy curve is the honest way to compare two positioning systems. Which of your two synthetic systems would you ship for a use case that tolerates rare large errors but needs a tight median, and which for one that forbids large errors entirely?
Self-check
- Why can two systems have identical CEP50 yet wildly different R95, and which underlying error behavior does a large CEP-to-R95 ratio reveal?
- What does a protection level bound that the corresponding accuracy figure does not, and what does a point below the diagonal on a Stanford diagram represent?
- A fingerprinting system reports 1.5 meter median accuracy on a random 80/20 scan split. Give a concrete reason this number is likely optimistic and the splitting change that would make it honest.
What's Next
In Section 25.7, we turn from measuring how good a location fix is to reckoning with what a location fix reveals: the privacy exposure of continuous positioning, the ease of re-identifying a person from a supposedly anonymous trajectory, and the spoofing and jamming attacks that let an adversary feed a receiver a confident but entirely fabricated position.