Streaming evaluation and prequential metrics

"My dashboard says 94% accuracy. It has said 94% since March. It is the lifetime average of a model that has been failing since June, and the good early days are burying the bad new ones."

A Complacent AI Agent

This section assumes the leakage discipline of Chapter 5 and the offline evaluation protocols of Chapter 65, which this extends into the online regime. It pairs naturally with the drift detectors of Section 60.5: a drift detector watches the input or the error stream for a change point, while a prequential metric tells you what the error actually is right now. We build the metric here; the two together form the monitoring layer of the streaming service.

Test-then-train: the prequential principle

The foundational idea, due to Dawid, is prequential (predictive-sequential) evaluation, and it is beautifully simple. For each arriving sample, first predict with the current model and record whether the prediction was right, then learn from the labeled sample and move on. Every sample is thus a test sample before it is ever a training sample, which means no example is scored by a model that has already seen it. This is the streaming analogue of a train-test split, and it is automatically leakage-safe by construction: causality is enforced by the arrow of time itself. Formally, if \(\hat{y}_t\) is the prediction made from state \(\theta_{t-1}\) and \(\ell\) is a per-sample loss, the prequential loss is the running aggregate

$$ L_t = \sum_{i=1}^{t} \ell\big(y_i, \hat{y}_i\big), \qquad \hat{y}_i = f_{\theta_{i-1}}(x_i), $$

and the reported metric is some average of the per-sample terms. The subtlety, and the entire content of this section, is which average. The naive choice, dividing \(L_t\) by \(t\), gives the cumulative mean, and that is the trap in the epigraph.

Windowed and fading estimators

Two forgetting schemes cover almost everything. A sliding-window estimator keeps the last \(w\) labeled predictions in a ring buffer and reports the mean over exactly those, so the metric reflects a crisp, interpretable horizon ("accuracy over the last 1000 samples") and drops old evidence completely. Its cost is a buffer of size \(w\) and a slight steppiness as samples enter and leave. A fading-factor (exponentially weighted) estimator instead decays the running sums geometrically with a factor \(\gamma \in (0,1)\), holding no buffer at all:

$$ S_t = \gamma\, S_{t-1} + \ell(y_t,\hat{y}_t), \qquad N_t = \gamma\, N_{t-1} + 1, \qquad \overline{\ell}_t = \frac{S_t}{N_t}. $$

Here \(N_t\) is the effective sample count, which converges to \(1/(1-\gamma)\); that quantity is the estimator's memory in samples, the direct analogue of the window length \(w\). A \(\gamma\) of \(0.999\) remembers about 1000 samples, \(0.9999\) about 10,000. The normalization by \(N_t\) matters: it makes the estimate unbiased from the very first sample, so you get a usable reading during warm-up instead of a figure creeping up from zero. Fading factors are the default in streaming libraries because they cost two floats and one multiply per sample and degrade gracefully, whereas a window trades memory for a sharper horizon. Choose the horizon from how fast you need to detect a regression: a safety-critical automotive perceptor may want a few hundred samples, a slow-degrading industrial asset tens of thousands.

class PrequentialMetric:
    "Test-then-train accuracy with either a sliding window or a fading factor."
    def __init__(self, gamma=0.999):
        self.gamma = gamma          # closer to 1 == longer memory
        self.correct = 0.0          # faded sum of correctness
        self.total = 0.0            # faded effective count

    def update(self, y_true, y_pred):
        # 1) TEST: score the prediction the model already committed to
        hit = 1.0 if y_true == y_pred else 0.0
        self.correct = self.gamma * self.correct + hit
        self.total   = self.gamma * self.total + 1.0
        # (2) TRAIN happens outside, AFTER this call: model.learn_one(x, y_true)
        return self.correct / self.total   # live accuracy, unbiased from t=1

# usage on a labeled stream
metric = PrequentialMetric(gamma=0.999)   # ~1000-sample memory
for x, y in stream:
    y_hat = model.predict_one(x)          # predict FIRST
    live_acc = metric.update(y, y_hat)    # then score
    model.learn_one(x, y)                 # then learn
    if live_acc < 0.80:
        raise_alarm(live_acc)             # act on the recent number

The code above makes the test-then-train order explicit and shows the fading estimate feeding an alarm threshold directly. Notice there is no train-test split anywhere: the stream is the split, unrolled in time.

Choosing the right metric, not just the right memory

Forgetting fixes when you measure; you still must choose what to measure, and the streaming setting sharpens the classic warnings. Sensor event streams are usually severely imbalanced: falls, arrhythmias, machine faults, and intrusions are rare by design, so a model that predicts "normal" forever scores 99% prequential accuracy while catching nothing. On a stream this is more dangerous than offline, because the positive class is not just rare but clustered in time: you can go hours between events, during which accuracy is a flat, reassuring, meaningless line. Track a metric that only moves when it should, such as rolling recall on the positive class, rolling F1, or a rolling cost-weighted error that prices a missed fault against a false alarm. For probabilistic and calibrated outputs (Chapter 18), a prequential log-loss or Brier score reacts to confidence drift before the hard-label accuracy does, giving earlier warning of a model losing its grip.

Delayed and missing labels: the hard part in practice

Prequential evaluation quietly assumes the label \(y_t\) is available right after the prediction. On real sensor systems it rarely is. A human activity recognizer may wait until a subject annotates a diary at day's end; a fault predictor's ground truth arrives only when a technician opens the machine weeks later; many streams never get a label at all. This label latency breaks naive test-then-train, because you cannot both alarm now and wait a week for truth. Three practices keep evaluation honest. First, buffer predictions with their timestamps and score them only when the matching label lands, so a prediction made Monday and confirmed Friday updates Friday's metric against Monday's model state, not a later one. Second, when labels are merely delayed by a known horizon \(h\), report the metric lagged by \(h\) and say so on the dashboard, rather than pretending the latest point is settled. Third, when labels are sparse or absent, fall back to unsupervised proxies: track the drift statistics of Section 60.5 on the input and on the model's output-confidence distribution, which need no labels and often move before accuracy does. A monitoring layer that mixes a lagged supervised metric with an always-on unsupervised proxy degrades gracefully when ground truth is scarce.

What's Next

A live metric that drops is a signal, not a resolution. In Section 60.7, we turn the monitoring curve into action: how to design incident handling for a streaming service, from alarm thresholds and hysteresis that avoid flapping, through automatic fallback to a safe model, to the human-in-the-loop escalation and rollback that close the loop when a deployed model has genuinely gone bad.