Monitoring risk during adaptation

"I minimized my entropy to zero and felt magnificent. I was now certain, calm, and predicting 'walking' for a man who had been asleep for six hours."

A Confidently Collapsed AI Agent

The failure modes an unsupervised update invites

Three collapse patterns account for most field incidents with test-time adaptation, and each is invisible to the adaptation loss itself. The first is entropy collapse. Entropy minimization, the engine of TENT, is minimized perfectly by a network that outputs the same one-hot vector for every input. On a benign stream that trivial solution is far away, but a burst of ambiguous or out-of-distribution windows, a sensor unplugged, a person out of view, hands the optimizer a shortcut: shrink the output entropy by pushing everything toward the currently dominant class. The loss drops, the confidence rises, and the classifier is now a constant function.

The second is class-prior collapse, a subtler cousin. The model does not freeze on one class but drifts its predicted marginal far from any plausible prior, over-calling the majority activity because doing so lowers batch entropy on average. The third is error accumulation in continual adaptation. CoTTA and its relatives update indefinitely; small biases compound across hours or days until the weights have wandered out of the basin they were trained in, a runtime form of catastrophic forgetting. The common thread is that the quantity being optimized (low entropy, temporal consistency) and the quantity we care about (accuracy) come apart under shift, and only the first is observable at test time. That gap is exactly what a monitor must span.

Label-free risk signals that actually move under failure

With no ground truth on the live stream, monitoring reduces to choosing proxies whose behavior under healthy adaptation is distinguishable from their behavior under collapse. Four earn their place. Predicted-class diversity is the sharpest early warning: track the running histogram of \(\arg\max\) predictions over a window and compute its entropy \(H(\hat p)\); when the model is collapsing onto one class, \(H(\hat p)\) falls toward zero even as per-sample confidence rises. It is the mirror image of the training signal, which is what makes it trustworthy.

Anchor disagreement keeps a frozen copy \(f_0\) of the shipped model and measures the divergence between the adapting model \(f_t\) and the anchor on the same windows, for example the symmetric KL between their softmax outputs. A small, stable disagreement means the adapter is refining; a rising trend means it is wandering. Because the anchor never updates, it is an incorruptible reference frame. Confidence calibration drift reuses Chapter 18's tools: a healthy adapter tends to stay near its shipped reliability, while a collapsing one becomes sharply overconfident, so a running gap between mean confidence and any available proxy accuracy is a red flag. Finally, feature-statistic drift watches the batch-norm running means and variances the adapter is allowed to touch; a sudden excursion of these statistics beyond the range seen in calibration is often the first physical sign that the input stream itself, not just the labels, has moved.

From signals to a trigger: control charts, filtering, and reset

A raw signal is not a decision. To act, wrap each proxy in the change-detection machinery of Chapter 12: maintain an exponentially weighted moving average of the diversity entropy and the anchor KL, and fire when either crosses a control limit set from a healthy calibration run, or when a CUSUM statistic accumulates enough evidence of a sustained shift. The response is tiered. The cheapest is sample filtering: refuse to adapt on windows whose per-sample entropy is above a reliability threshold, so ambiguous and out-of-distribution inputs never get to steer the weights. This is the core idea of EATA (Niu and colleagues, ICML 2022), which pairs entropy-based sample selection with a Fisher-weighted anti-forgetting regularizer that anchors weights important to the original task. The next tier is rollback: on a fired alarm, restore the adaptable parameters to the last checkpoint that passed the monitor, the runtime equivalent of a circuit breaker. The strongest tier is freeze: disable adaptation entirely and fall back to the frozen anchor for the rest of the session, trading the upside of adaptation for a guaranteed floor.

import copy, torch
import torch.nn.functional as F

class AdaptationMonitor:
    """Label-free watchdog for a test-time-adapting classifier."""
    def __init__(self, model, n_classes, div_floor=0.35, kl_ceil=0.8, beta=0.1):
        self.anchor = copy.deepcopy(model).eval()   # frozen reference f_0
        self.n = n_classes
        self.div_floor, self.kl_ceil, self.beta = div_floor, kl_ceil, beta
        self.hist = torch.ones(n_classes) / n_classes  # running class marginal
        self.kl_ema = 0.0
        self.checkpoint = copy.deepcopy(model.state_dict())

    def check(self, model, x):
        with torch.no_grad():
            p  = F.softmax(model(x), dim=1)           # adapting model f_t
            p0 = F.softmax(self.anchor(x), dim=1)     # anchor f_0
        # 1. predicted-class diversity (normalized to [0,1])
        self.hist = (1 - self.beta) * self.hist + self.beta * p.mean(0)
        diversity = -(self.hist * self.hist.clamp_min(1e-9).log()).sum() / \
                    torch.log(torch.tensor(float(self.n)))
        # 2. anchor disagreement (symmetric KL, EMA-smoothed)
        kl = 0.5 * (F.kl_div(p.log(), p0, reduction='batchmean') +
                    F.kl_div(p0.log(), p, reduction='batchmean'))
        self.kl_ema = (1 - self.beta) * self.kl_ema + self.beta * kl.item()
        collapsing = (diversity < self.div_floor) or (self.kl_ema > self.kl_ceil)
        return collapsing, float(diversity), self.kl_ema

    def rollback(self, model):                        # circuit breaker
        model.load_state_dict(self.checkpoint)

Code 66.6.1 is the concrete form of the two most reliable proxies from the previous section, combined into one guard the inference loop calls each batch. In practice you checkpoint whenever the monitor is green for a sustained stretch and call rollback the instant it goes red, so the adapter can improve during good weather yet never spend more than a window's worth of drift. SAR (Niu and colleagues, ICLR 2023) formalizes the same instinct at the optimizer level: it filters high-gradient noisy samples and uses sharpness-aware minimization to keep the adapted weights in a flat, stable region, plus an explicit model-reset when a moving average of the loss signals collapse.

Guarantees, guardrails, and knowing when to stop

Monitoring proxies raise alarms; they do not by themselves bound risk. Where the cost of a wrong action is real, wrap the adapting predictor in the distribution-free machinery of Chapter 18. Conformal prediction sets, recalibrated on a rolling window, turn a collapsing model into a self-evident one: as calibration degrades, the prediction sets inflate, and a set that swells past a usable size becomes a hard signal to abstain or defer rather than act on a confident guess. This is the runtime-monitoring and safety-envelope stance that Chapter 68 makes mandatory for safety-relevant perception: adaptation is permitted only inside an envelope, and the moment the monitor says the model has left it, the system reverts to a certified frozen fallback.

The operational picture belongs to the fleet. Every alarm, rollback, and freeze is a proxy-quality event that Chapter 69's monitoring under delayed ground truth aggregates: if the same device rolls back every shift, adaptation is not the fix, retraining or recall is. The final judgment a monitor supports is negative capability, knowing when to stop adapting. A model that has converged on a stable target should freeze and bank the gain; continuing to adapt only exposes it to the next out-of-distribution burst with nothing left to lose. The best adapters spend most of their life not adapting, watched by a watchdog that is cheap, incorruptible, and always allowed to say no.

What's Next

In Section 66.7, we leave the runtime and return to measurement: the benchmarks, HAROOD, BenchHAR, and WILDS, that let us quantify all of this chapter's claims about shift, OOD detection, generalization, and adaptation under leakage-safe, reproducible conditions, so that the recovery an adapter reports is a real gain and not a split artifact.