Conformal prediction: split, weighted, and the exchangeability problem

"You asked for a promise, not a hunch. I can give you a set that catches the truth ninety percent of the time, but only if you swear the future looks like the calibration data. It rarely does."

A Contractually Cautious AI Agent

Split conformal in one pass

Split (or inductive) conformal prediction is disarmingly simple. Partition your data into a training fold and a calibration fold that the model never sees during fitting. Train any model \(\hat{f}\) on the training fold. Then define a nonconformity score \(s(x, y)\) that is large when the pair \((x,y)\) looks surprising to the model. For regression the natural choice is the absolute residual \(s = |y - \hat{f}(x)|\); for classification a common choice is \(s = 1 - \hat{p}(y \mid x)\), the model's assigned improbability of the true class. Compute the scores \(s_1, \dots, s_n\) on the \(n\) calibration points, then take the quantile

$$\hat{q} = \text{the } \lceil (n+1)(1-\alpha) \rceil \text{-th smallest of } \{s_1, \dots, s_n\}.$$

At test time, output every label whose score falls at or below the threshold: \(\mathcal{C}(x) = \{\, y : s(x,y) \le \hat{q}\,\}\). For regression this is the fixed-width band \(\hat{f}(x) \pm \hat{q}\); for classification it is a set of plausible classes, sometimes one, sometimes several when the model is unsure. The theorem is that, if the calibration and test points are exchangeable, the set covers the truth with probability at least \(1-\alpha\), and at most \(1-\alpha+1/(n+1)\). Notice the finite-sample correction \((n+1)\) in the quantile rank: it is what turns an asymptotic promise into an exact one, and it is why conformal coverage holds even for a calibration set of a few hundred points.

The code below runs the entire split conformal procedure for a regression predictor on a sensor target and empirically checks the coverage, the number you must always verify rather than trust.

import numpy as np

def split_conformal_band(cal_pred, cal_true, alpha=0.1):
    scores = np.abs(cal_true - cal_pred)          # nonconformity: absolute residual
    n = len(scores)
    rank = int(np.ceil((n + 1) * (1 - alpha)))    # finite-sample corrected rank
    rank = min(rank, n)                           # guard when (n+1)(1-a) > n
    return np.sort(scores)[rank - 1]              # the conformal quantile q_hat

# cal_* is the calibration fold, test_* is a disjoint test fold
q = split_conformal_band(cal_pred, cal_true, alpha=0.1)
lo, hi = test_pred - q, test_pred + q
covered = (test_true >= lo) & (test_true <= hi)
print(f"target coverage 0.90 | empirical {covered.mean():.3f} | half-width {q:.3f}")

The exchangeability assumption, and why streams break it

A sequence of random variables is exchangeable if its joint distribution is invariant to reordering: any permutation of the samples is equally probable. This is weaker than independent and identically distributed, but it is still a symmetry assumption, and it is the single hinge the coverage proof hangs on. Sensor time series violates it in at least three structural ways. First, temporal dependence: consecutive samples from an accelerometer, an ECG, or a vibration probe are strongly autocorrelated, so a window and its neighbor are not swappable with a distant window. Second, distribution shift: the test stream is collected later than, and often on different hardware or a different subject from, the calibration stream, so the marginal distribution of inputs drifts, a problem taken up in full in Chapter 66. Third, non-stationarity: sensor baselines wander with temperature, battery, wear, and calibration age, as the measurement models of Chapter 2 make explicit. When exchangeability fails, split conformal does not merely lose a little coverage; its empirical coverage can collapse well below \(1-\alpha\) exactly when you needed the guarantee most, during the drift.

Weighted conformal prediction under covariate shift

One failure mode is repairable in closed form. Suppose only the input distribution changes, from \(P_{\text{cal}}(x)\) to \(P_{\text{test}}(x)\), while the conditional \(P(y \mid x)\) stays fixed. This is covariate shift, and it describes much of sensor deployment: the same physical relationship between signal and label, measured on a new population of inputs. Weighted conformal prediction (Tibshirani and colleagues, 2019) restores validity by reweighting each calibration score by the likelihood ratio \(w(x) = P_{\text{test}}(x) / P_{\text{cal}}(x)\). Instead of an unweighted quantile, you take the quantile of the calibration scores under the normalized weights

$$p_i = \frac{w(x_i)}{\sum_{j=1}^{n} w(x_j) + w(x_{\text{test}})},$$

where the test point itself contributes a term because its own weight enters the tie-broken rank. Calibration points that resemble the test distribution count more; those from stale regions count less. The insight is that exchangeability is not gone, it is only tilted, and a known tilt can be undone by a known weight. The practical difficulty moves entirely into estimating \(w(x)\), typically with a probabilistic classifier trained to discriminate calibration inputs from test inputs, which is itself an imperfect model whose error erodes the guarantee. Weighted conformal buys back coverage under covariate shift at the price of a density-ratio estimate, and it says nothing about a shift in \(P(y \mid x)\), which needs the adaptive machinery of the next section.

from mapie.regression import MapieRegressor
mapie = MapieRegressor(estimator=my_fitted_model, cv="prefit")
mapie.fit(X_cal, y_cal)                       # calibration fold only
y_pred, y_pis = mapie.predict(X_test, alpha=0.1)   # 90% intervals

What's Next

In Section 18.5, we drop the exchangeability assumption altogether. Adaptive conformal inference and its conformal-PID refinement treat coverage as a control problem, nudging the threshold up or down after every observation so that a drifting, non-stationary sensor stream still hits its long-run coverage target without any weight estimate or stationarity promise.