Calibration and temperature scaling for time series

"When I say I am 90 percent sure, I want to be wrong exactly one time in ten. A confident liar is worse than an honest doubter."

A Well-Calibrated AI Agent

What calibration means, and how to measure it

Let a classifier output a probability vector \(\hat{p}(x)\) with predicted class \(\hat{y}=\arg\max_k \hat{p}_k(x)\) and confidence \(\hat{c}=\max_k \hat{p}_k(x)\). Perfect calibration is the statement

$$\mathbb{P}\big(\hat{y}=y \,\big|\, \hat{c}=c\big) = c \quad\text{for all } c\in[0,1].$$

Read literally: among all windows the model called "80 percent," it should be right 80 percent of the time. We cannot condition on a continuous \(c\), so we bin. Sort predictions into \(M\) confidence bins \(B_1,\dots,B_M\), and compare each bin's average confidence to its empirical accuracy. The Expected Calibration Error is the weighted average gap,

$$\mathrm{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{n}\,\big|\operatorname{acc}(B_m) - \operatorname{conf}(B_m)\big|,$$

and plotting \(\operatorname{acc}(B_m)\) against \(\operatorname{conf}(B_m)\) gives the reliability diagram: the diagonal is perfect, points below it are overconfident (the usual failure), points above are underconfident. ECE is a diagnostic, not a training loss; it is not a proper scoring rule and can be gamed. Report it alongside negative log-likelihood or Brier score, which are proper (Chapter 4), so that improving calibration cannot secretly wreck sharpness.

Temperature scaling: one number, learned after training

Train the network normally. Freeze it. Now introduce a single scalar \(T>0\), the temperature, and soften every logit vector \(z(x)\) before the softmax:

$$\hat{p}_k(x;T) = \frac{\exp\big(z_k(x)/T\big)}{\sum_j \exp\big(z_j(x)/T\big)}.$$

With \(T=1\) nothing changes. \(T>1\) flattens the distribution toward uniform (less confident); \(T<1\) sharpens it. Because \(T\) scales all logits equally, the argmax and hence every predicted label is invariant. You fit \(T\) by minimizing negative log-likelihood on a held-out calibration set that the network never trained on, a one-dimensional convex-in-practice optimization solved in a handful of iterations. This is Guo et al.'s 2017 result: across modern architectures a single temperature recovers near-perfect calibration, outperforming heavier fixes like Platt scaling, isotonic regression, or Bayesian binning on the metric that matters, while adding exactly one parameter. The code below fits \(T\) on cached logits.

import numpy as np
from scipy.optimize import minimize_scalar

def temperature_scale(logits, labels):
    # logits: (n, K) raw pre-softmax outputs on the CALIBRATION split
    # labels: (n,) integer class ids
    def nll(T):
        z = logits / T
        z = z - z.max(axis=1, keepdims=True)          # stable softmax
        logp = z - np.log(np.exp(z).sum(axis=1, keepdims=True))
        return -logp[np.arange(len(labels)), labels].mean()
    res = minimize_scalar(nll, bounds=(0.05, 10.0), method="bounded")
    return res.x

def ece(probs, labels, n_bins=15):
    conf = probs.max(axis=1); pred = probs.argmax(axis=1)
    correct = (pred == labels).astype(float)
    bins = np.linspace(0, 1, n_bins + 1)
    e = 0.0
    for lo, hi in zip(bins[:-1], bins[1:]):
        m = (conf > lo) & (conf <= hi)
        if m.any():
            e += m.mean() * abs(correct[m].mean() - conf[m].mean())
    return e

T = temperature_scale(cal_logits, cal_labels)         # e.g. T = 1.9
scaled = np.exp(cal_logits / T)
scaled /= scaled.sum(axis=1, keepdims=True)
print(f"T={T:.2f}  ECE {ece(softmax(test_logits), test_labels):.3f} "
      f"-> {ece(softmax(test_logits / T), test_labels):.3f}")

Why time series breaks the textbook recipe

Every guarantee above rests on one assumption: the calibration split is a representative, exchangeable sample of what test time looks like. For a shuffled image benchmark that is nearly free. For a sensor stream it is where projects fail silently. Three specific traps:

• Autocorrelation inflates your effective calibration set. Ten thousand consecutive accelerometer windows from one afternoon are not ten thousand independent samples; they may be worth a few dozen. A temperature fit on them is fit to that afternoon's operating point, and its ECE estimate is optimistically tiny. Calibrate on windows spanning many sessions, subjects, and devices, and evaluate calibration with a block or subject-wise split, never a random per-window shuffle (the leakage rule of Chapter 5).
• Drift makes a fixed \(T\) stale. A sensor that ages, a patient whose baseline shifts, a machine entering a new load regime: the logit distribution moves, and the temperature that was honest in March is overconfident by September. Temperature scaling is a static, one-shot repair; under genuine distribution shift you need the adaptive machinery of Chapter 66 and the online conformal methods of Section 18.5.
• Step-level versus sequence-level. A model emitting a probability every 20 ms produces highly dependent confidences; calibrating each timestep independently ignores that a whole event (one fall, one arrhythmia episode) is the real decision unit. Calibrate at the granularity your downstream consumer acts on.

Calibrating regression and forecast intervals

Most sensor deep learning is regression or forecasting, not classification, and "calibration" generalizes cleanly. A probabilistic forecaster emits quantiles or a predictive distribution; it is calibrated if its stated coverage matches reality. Concretely, for every level \(\alpha\), the fraction of test targets falling below the predicted \(\alpha\)-quantile should equal \(\alpha\), and a 90 percent prediction interval should contain the truth 90 percent of the time. The regression analogue of the reliability diagram plots nominal coverage against empirical coverage; the analogue of temperature scaling is a learned rescaling of the predicted standard deviation \(\hat{\sigma}\to s\,\hat{\sigma}\), or a monotone recalibration map fit on held-out residuals (Kuleshov et al., 2018). The same three time-series traps apply, sharpened: interval calibration under autocorrelation and drift is exactly the pressure that motivates conformal prediction.

What's Next

In Section 18.3, we stop repairing a single point estimate after the fact and instead ask the model to represent its own uncertainty from the start: deep ensembles, MC-dropout, Bayesian layers, and evidential deep learning. Temperature scaling fixed the confidence you already had; those methods try to produce a trustworthy distribution in the first place, and they capture the epistemic gap that a lone temperature, however honest, cannot see.