Adaptive conformal inference (ACI) and conformal PID under drift

"You promised me ninety percent coverage. The sensor promised me a stationary world. One of you lied, and I have to pick which knob to turn."

A Feedback-Minded AI Agent

This section assumes you have the split-conformal machinery and the exchangeability failure from Section 18.4, the notion of a nonconformity score, and the coverage-versus-width tradeoff. It also leans on the drift and change-detection vocabulary from Chapter 12 and the online-learning setting of Chapter 60. If the probabilistic framing of coverage feels thin, the primer in Chapter 4 is the place to shore it up.

Why a frozen threshold fails on a moving stream

Recall the split-conformal recipe. You hold out a calibration set, compute nonconformity scores \(s_i = s(x_i, y_i)\), and take the threshold \(\hat q\) as the \(\lceil (n+1)(1-\alpha)\rceil / n\) empirical quantile. The prediction set is \(C(x) = \{y : s(x,y) \le \hat q\}\). Marginal coverage \(\mathbb{P}(y \in C(x)) \ge 1-\alpha\) holds because the test score is exchangeable with the calibration scores, so its rank is uniform.

Drift breaks the rank argument. Suppose a vibration model was calibrated on a bearing at 40 degrees Celsius and the machine now runs at 70 degrees. Residuals grow, scores shift right, and the fraction of test points exceeding the old \(\hat q\) climbs past \(\alpha\). Coverage decays exactly when the operating condition is most unusual, which is the worst possible time to be over-confident. Re-calibrating on a sliding window helps but chases a target it never catches, and it needs fresh ground-truth labels that a deployed sensor rarely has in real time. We want a guarantee that survives arbitrary, even adversarial, distribution change.

Adaptive conformal inference as an integral controller

The idea from Gibbs and Candes (2021) is to make the target level itself adaptive. Keep a state variable \(\alpha_t\), form the set \(C_t\) at the \(1-\alpha_t\) quantile of recent scores, observe whether the truth landed inside, and nudge \(\alpha_t\) by the coverage error. Define the miscoverage indicator \(\mathrm{err}_t = \mathbf{1}[y_t \notin C_t]\). The update is a single line:

When you miss (\(\mathrm{err}_t = 1\)), \(\alpha_{t+1}\) drops, so next step demands a higher quantile and a wider interval. When you cover (\(\mathrm{err}_t = 0\)), \(\alpha_{t+1}\) rises and the interval tightens. The step size \(\gamma\) is the learning rate. This is precisely integral control: \(\alpha_t\) accumulates the running coverage error and drives it to zero. Summing the recursion telescopes into a clean bound. For scores in a bounded range, the realized miscoverage satisfies

so the time-averaged coverage converges to \(1-\alpha\) at rate \(O(1/T)\), for every input sequence, with no distributional assumption whatsoever. Larger \(\gamma\) tracks abrupt shifts faster but makes the width jittery; smaller \(\gamma\) is smooth but sluggish after a regime change. A common default is \(\gamma \approx 0.005\) to \(0.05\), tuned so the width settles within your acceptable reaction time.

Conformal PID: proportional and derivative terms

ACI is the integral term alone, and integral-only control has a known weakness: it lags. Under a steady trend (scores creeping up week over week), an integrator is always a step behind, so it chronically under-covers by a small margin. Angelopoulos, Candes, and Tibshirani (2024) reframe the whole thing as PID control of the conformal threshold and add the missing terms. Rather than steering \(\alpha_t\), they directly track the score quantile \(q_t\) with three contributions:

The integral term is ACI in a saturating wrapper \(r_t\) that keeps the state from winding up during long violations. The proportional term reacts to the instantaneous error for faster settling. The derivative term is the interesting one: instead of a naive difference, it is a scorecaster, a small forecaster (even the same class of model you use for the signal, per Chapter 16) that predicts next step's conformal score \(g_{t+1}\). When the scorecaster sees a rising residual trend it widens the interval before the misses accumulate, giving lead compensation. Crucially, the same long-run coverage guarantee holds regardless of how good or bad the scorecaster is: a useless scorecaster degrades gracefully to plain ACI, while an accurate one sharply reduces the transient under-coverage after a change point.

import numpy as np

def aci_stream(scores, y_in_set_fn, alpha=0.1, gamma=0.02, window=200):
    """Online ACI over a drifting score stream. Returns per-step coverage flags."""
    alpha_t, covered = alpha, []
    buf = []  # recent scores used as the running calibration set
    for t, s_t in enumerate(scores):
        if len(buf) >= 20:                      # need some history to form a quantile
            level = np.clip(1.0 - alpha_t, 0.0, 1.0)
            q_t = np.quantile(buf, level)        # adaptive threshold
            hit = y_in_set_fn(t, q_t)            # did truth fall inside C_t?
        else:
            hit = True
        covered.append(hit)
        err_t = 0.0 if hit else 1.0
        alpha_t = alpha_t + gamma * (alpha - err_t)   # integral update
        buf.append(s_t)
        if len(buf) > window:
            buf.pop(0)
    return np.array(covered)

The loop above shows the mechanism, and it is worth running once to feel how \(\gamma\) trades jitter against lag. In production you would not hand-roll the buffer, the winsorized quantile, and the PID saturation.

Choosing knobs and reading the width signal

Three practical rules carry most of the value. First, tune \(\gamma\) (and the proportional \(\eta\)) to the fastest drift you must survive, not the average one: pick the shortest change-point-to-recovery time you can tolerate and set the gain so the width settles inside it. Second, treat the interval width itself as a first-class monitoring signal. A width that is quietly ballooning is the controller telling you the model has drifted out of its competence; that is a maintenance trigger and a natural handoff to the abstention logic of Section 18.6. Third, remember the guarantee is average, not conditional: ACI can hold 90 percent overall while systematically missing one rare regime. If per-condition coverage matters (a safety case for a specific fault mode), combine adaptive updates with the conditional or Mondrian conformal ideas from the previous section, and validate under the distribution-shift protocols of Chapter 66.

What's Next

In Section 18.6, we turn a ballooning conformal interval into a decision: when the width or the drift signal says the model is out of its depth, the system should abstain, fall back to a conservative estimate, or hand off to a human. We build the abstention thresholds and safety margins that sit on top of the adaptive coverage machinery from this section.