Proxy monitoring under delayed ground truth

"They asked me for my accuracy. The labels that would tell me my accuracy arrive in three weeks, for one batch in ten, and only when a human bothered to look. So I told them about my confidence instead, and I told them how much I trusted my confidence, and that turned out to be the honest answer."

A Provisionally Correct AI Agent

The three ways ground truth fails to show up on time

It is worth being precise, because each failure mode calls for a different remedy. Latency is the gap between a prediction at time \(t\) and the label at time \(t + \Delta\). For a maintenance model \(\Delta\) can be months; for a clinical over-read, weeks; for a click on a recommended action, minutes. A monitor that reports accuracy is always reporting the accuracy of predictions made \(\Delta\) ago, which is useless for catching a regression that started yesterday. Censoring means labels arrive for only a subset of predictions: a device is retired before its failure is observed, a patient never returns, a flagged event is never adjudicated. The label is not late; it is absent, and treating absent as negative silently corrupts every rate you compute. Verification bias is the sharpest trap: the labels that do arrive are not a random sample. A human investigates the alarms the model raised, so you learn the true state of positives far more often than negatives. Compute precision on that trickle and it looks plausible; compute recall and it is a fantasy, because the false negatives were never looked at. A proxy-monitoring system has to be designed around all three at once.

A taxonomy of proxy signals

Proxies fall into four families, ordered from cheapest to most trustworthy. Confidence-based proxies read the model's own output distribution: mean maximum-softmax probability, predictive entropy, or conformal set size (Chapter 18). If the model is calibrated, average confidence is a direct, if noisy, estimate of average accuracy. Distributional proxies watch the prediction distribution rather than the input: the flagged-anomaly rate, the class mix, the score histogram. A silent doubling of the positive rate is a five-alarm signal even with zero labels, because the base rate of real failures did not double overnight. Consistency proxies exploit redundancy: agreement between an ensemble's members, between two sensors that should see the same event (the fusion redundancy of Chapter 48), or between the current model and the previous version run in shadow. Disagreement needs no ground truth and rises sharply under shift. Behavioral and downstream-outcome proxies are the most trustworthy because they touch reality: the human-override rate (how often an operator dismisses the model's alarm), the intervention outcome (did the dispatched crew find a real fault), the complaint rate. These carry real signal about correctness but arrive with their own latency, so they close the loop slowly. A mature fleet monitor tracks one or two proxies from each family and cross-checks them, because any single proxy can be fooled.

Turning a proxy into an estimate of the real metric

Watching a proxy wiggle is monitoring; turning it into a number you can put a threshold on is estimation. The workhorse for classification is Average Thresholded Confidence (ATC). The idea: on a labeled reference set, find a threshold \(t\) on a confidence score \(s(x)\) (for example the maximum softmax probability, or negative entropy) such that the fraction of reference points scoring below \(t\) equals the reference error rate. Formally, choose \(t\) so that

Then, on a fresh unlabeled batch from the fleet, estimate the error as simply the fraction of points scoring below that same \(t\). This works because a calibrated score orders points by correctness the same way in the reference and target distributions under a broad shift assumption; it converts a whole confidence histogram into one accuracy number without a single target label. Direct-loss-estimation and importance-weighted variants extend the same trick to AUC and to regression. Critically, ATC must be re-validated whenever backfilled labels arrive: each time a batch of ground truth lands, you compare the estimate the proxy produced weeks ago against the truth, and if the residual is growing you re-fit \(t\) or retire the proxy. That backfill loop is what keeps the estimate honest as the fleet drifts.

import numpy as np

def fit_atc_threshold(conf_ref, correct_ref):
    """Pick threshold t so P(conf < t) equals the reference error rate."""
    err_rate = 1.0 - correct_ref.mean()          # measured on labeled reference
    t = np.quantile(conf_ref, err_rate)          # the err_rate-quantile of confidence
    return t

def estimate_error(conf_target, t):
    """Label-free error estimate on an unlabeled fleet batch."""
    return float(np.mean(conf_target < t))

rng = np.random.default_rng(0)
# Reference: calibrated model, ~12% error. Confidence higher when correct.
correct_ref = rng.random(5000) > 0.12
conf_ref = np.clip(rng.normal(np.where(correct_ref, 0.85, 0.55), 0.12), 0, 1)
t = fit_atc_threshold(conf_ref, correct_ref)

# Fleet batch after a drift that pushed true error to ~22% (labels unavailable).
correct_true = rng.random(5000) > 0.22
conf_tgt = np.clip(rng.normal(np.where(correct_true, 0.80, 0.50), 0.14), 0, 1)
print(f"threshold t          = {t:.3f}")
print(f"estimated error      = {estimate_error(conf_tgt, t):.3f}")
print(f"true error (oracle)  = {1 - correct_true.mean():.3f}")

Correcting the labels you do get

Eventually some ground truth backfills, and it is tempting to just score against it. Do not, before you correct for how it was selected. If humans only adjudicate model-flagged positives, the labeled set is a biased sample and naive recall is uncomputable, since the false negatives are invisible. Three practices help. First, deliberate label sampling: budget for a small random audit of unflagged cases so you can estimate the false-negative rate, not just the false-positive rate; a stratified draw of a few hundred negatives per month often costs less than one missed failure. Second, importance weighting: when the probability of a case being labeled depends on measurable features, weight each label by the inverse of its selection probability to recover an unbiased metric (the same reweighting machinery as label shift in Chapter 66). Third, delayed and prequential evaluation: report metrics on a lagged window whose labels are now mature, sliding forward as more arrive, and always show the fraction of a window's predictions that are still unlabeled so a reader knows how provisional the number is. A metric on 12% resolved labels is a rumor; the dashboard should say so.

Wiring it into the fleet monitor

Operationally, proxy monitoring is a second track running alongside the input-drift monitor of Section 69.3. Each batch emits its proxy vector (mean confidence, entropy, positive rate, ensemble disagreement, override rate) and an ATC-style error estimate with a confidence band. A regression fires when the estimate crosses a threshold and at least one independent proxy family agrees, which suppresses the false alarms that any single noisy proxy throws. Separately, a slow reconciliation job ingests backfilled labels, recomputes bias-corrected true metrics on now-mature windows, and audits every past proxy estimate against the truth; when the proxy-to-truth residual drifts, it schedules a re-fit or opens a ticket. That reconciliation is precisely the evidence a retraining trigger needs, which is where we turn next.

What's Next

In Section 69.5, we turn these proxy estimates and backfilled metrics into decisions: the retraining triggers that decide when a drift signal or a confirmed accuracy drop justifies retraining, and the fleet-management machinery that rolls a new model out across thousands of heterogeneous devices without taking the whole fleet down at once.