Calibration and uncertainty metrics

"I was ninety-five percent confident, right up until the moment I was wrong for the ninth time out of ten."

An Overconfident AI Agent

Calibration is a distinct axis from accuracy

Two models can share an identical AUROC and behave completely differently once deployed. Consider a fall detector that scores every window at either 0.49 or 0.51: its ranking is fine, its accuracy at threshold 0.5 may be excellent, yet its probabilities are meaningless because it never expresses genuine confidence. A well-calibrated classifier satisfies a simple property: among all inputs assigned confidence \(p\), the empirical fraction that are correct is also \(p\). Formally, for predicted confidence \(\hat{p}\) and correctness indicator, perfect calibration means \(\mathbb{P}(y = \hat{y} \mid \hat{p} = p) = p\) for all \(p \in [0,1]\).

Why insist on this on top of accuracy? Because sensing systems rarely act on the argmax alone. An industrial anomaly score feeds a cost-sensitive threshold; a fusion stage weights each sensor by its stated reliability (Chapter 49); a clinician reads the probability itself, not just the label. The probability calculus assumed by all of these is only valid if the numbers are calibrated. Calibration and accuracy are also traded against each other by modern training: deep networks trained with cross-entropy and heavy augmentation are frequently sharp but overconfident, so a rising accuracy curve can hide a worsening reliability curve unless you plot both.

Metrics for classification: reliability, ECE, and proper scores

The visual anchor is the reliability diagram: bin predictions by confidence, then plot mean confidence against mean accuracy per bin. The diagonal is perfect calibration; bars below it signal overconfidence, above it underconfidence. Reducing that picture to one number gives the Expected Calibration Error. Partition the \(N\) predictions into \(M\) confidence bins \(B_m\) and take a weighted average of the per-bin gap:

The Maximum Calibration Error replaces the sum with a max over bins, which matters when a single confidence region drives safety decisions. ECE has real pitfalls: it depends on the binning scheme, it can be gamed, and equal-width bins leave high-confidence bins nearly empty. Prefer adaptive (equal-mass) binning, report the bin count, and treat ECE as a summary of the diagram rather than a substitute for it.

Because ECE is not a proper scoring rule, always pair it with one. The Brier score is the mean squared error of probabilities, \(\frac{1}{N}\sum_i (\hat{p}_i - y_i)^2\), and decomposes cleanly into reliability, resolution, and uncertainty terms, separating "are the probabilities honest" from "are they informative." Negative log-likelihood (equivalently cross-entropy on the test set) is the other standard proper score; it punishes confident mistakes harshly, which is exactly the failure mode you fear in a safety context. A rule of thumb: report NLL or Brier as the headline uncertainty metric because they cannot be trivially gamed, and use ECE plus the reliability diagram to diagnose the direction and shape of the miscalibration.

Metrics for regression and predictive intervals

Much of sensory AI is regression: predicting heart rate from PPG, remaining useful life from vibration, or depth from a monocular frame. Here calibration is about intervals, not class probabilities. If a model emits a 90% prediction interval, the true value should fall inside it 90% of the time. Two metrics capture this directly. Prediction Interval Coverage Probability (PICP) is the empirical fraction of test targets inside their stated intervals; Mean Prediction Interval Width (MPIW) measures how tight those intervals are. As before, coverage without sharpness is trivial (an infinitely wide interval always covers), so read PICP and MPIW as a pair.

For probabilistic forecasts that emit a full predictive distribution, the Continuous Ranked Probability Score (CRPS) generalizes both absolute error and the Brier score to distributions and is a proper score, rewarding forecasts that are both accurate and appropriately sharp. A complementary check is quantile calibration: compute a calibration curve of nominal versus observed quantile coverage across many levels, the regression analogue of the reliability diagram. Conformal prediction, developed in Chapter 18, gives distribution-free coverage guarantees; this section supplies the metrics you use to verify that its promised coverage actually holds on your split.

Measuring calibration without leaking

Calibration metrics are only honest if the split discipline of this chapter is respected. Three failures recur. First, tuning on test: post-hoc recalibrators such as temperature scaling or isotonic regression fit a parameter on labeled data, so that fit must use a dedicated calibration split, never the test set whose ECE you will report. Second, subject leakage: if windows from one person appear in both the calibration and test folds, apparent calibration is inflated exactly as accuracy would be (Chapter 5). Third, calibration under shift: a model calibrated in the lab drifts as sensors age, users change, and seasons turn, so report calibration not only in-distribution but on time-forward and cross-site folds. A rising ECE across time-ordered folds is an early warning that feeds directly into the distribution-shift monitoring of Chapter 66. Finally, always attach a confidence interval to ECE and Brier via bootstrapping over subjects, not windows, so the uncertainty of your uncertainty metric respects the same grouping.

import numpy as np

def ece_and_reliability(probs, labels, n_bins=15):
    """Equal-mass ECE plus per-bin accuracy/confidence for a reliability diagram.
    probs: predicted confidence of the chosen class, shape (N,).
    labels: 1 if that prediction was correct, else 0, shape (N,)."""
    order = np.argsort(probs)
    probs, labels = probs[order], labels[order]
    bins = np.array_split(np.arange(len(probs)), n_bins)  # equal mass
    ece, rows = 0.0, []
    for b in bins:
        if len(b) == 0:
            continue
        conf = probs[b].mean()
        acc = labels[b].mean()
        ece += (len(b) / len(probs)) * abs(acc - conf)
        rows.append((conf, acc, len(b)))
    return ece, rows

rng = np.random.default_rng(0)
p = rng.uniform(0.5, 1.0, size=5000)          # stated confidence
correct = (rng.uniform(size=5000) < p - 0.08)  # 8% overconfident
print(f"ECE = {ece_and_reliability(p, correct.astype(int))[0]:.3f}")

The snippet above is deliberately explicit so the mechanics are visible, but you would not hand-roll it in production.

What's Next

In Section 65.7, we close the chapter by turning a single-run evaluation into a reproducible, field-validated protocol: seeds and environment capture, versioned data and code, pre-registered metric suites, and the bridge from a benchmark number to evidence that holds up in deployment and audit.