Covariate, label, and concept shift in sensing (builds on the uncertainty tools of Ch 18)

"They told me the world had changed. They would not tell me which part. So I corrected for the wrong one and grew confident about being wrong."

A Miscalibrated AI Agent

Three shifts, one factorization

Fix notation. A window of sensor features is \(x\), its label is \(y\), the training world is the source \(P_{\text{src}}\), and the field is the target \(P_{\text{tgt}}\). The reason we bother to distinguish sub-types of shift is that the joint distribution can be written two equivalent ways, and each way isolates a different assumption you might be willing to make:

A named shift is a claim that exactly one factor on one side moves and the other stays put. Covariate shift asserts \(P_{\text{tgt}}(x) \neq P_{\text{src}}(x)\) but \(P_{\text{tgt}}(y \mid x) = P_{\text{src}}(y \mid x)\). Label shift asserts \(P_{\text{tgt}}(y) \neq P_{\text{src}}(y)\) but \(P_{\text{tgt}}(x \mid y) = P_{\text{src}}(x \mid y)\). Concept shift asserts \(P_{\text{tgt}}(y \mid x) \neq P_{\text{src}}(y \mid x)\), the one factor everything else was trying to hold fixed. The value of the taxonomy is entirely practical: the invariant factor is the lever. If \(P(y \mid x)\) is invariant, your classifier is still correct on any \(x\) it sees and you only need to fix the input coverage. If \(P(x \mid y)\) is invariant, each class still "looks" the same and you only need to fix the class proportions. If neither is invariant, no amount of unlabeled target data can save you, because the mapping you learned is now the wrong mapping.

Covariate shift: the inputs are new, the rule is not

Covariate shift is the sensor practitioner's default hazard, because sensors live in the physical world and the physical world is not stationary. A new device revision changes the noise floor; an accelerometer worn tighter changes the amplitude envelope; ambient temperature drifts the bias of a MEMS gyro; a new deployment site has different vibration harmonics. In all of these the labeling rule is untouched: a fall is still a fall, a bearing fault still sounds like a bearing fault. Only the density of inputs \(P(x)\) has moved, often into regions the training set sampled thinly.

The classical correction is importance weighting. Because the risk under the target factors as an expectation under the source re-weighted by the density ratio,

you can reweight the training loss (or a validation estimate) by \(w(x)\) without ever labeling a target sample. You rarely estimate the two densities separately; the robust trick, echoing the domain-classifier idea, is to train a probe that discriminates source from target and read the ratio off its odds. The catch that Chapter 18 prepares you for: importance weights inflate variance when the two supports barely overlap, and a high weight on a region the model handles badly is exactly where your calibrated confidence should already be widening. Covariate shift and the change-detection machinery of Chapter 12 are two views of the same event: a monitored feature distribution moving off its baseline.

Label shift: the mix changed, the classes did not

Label shift is the mirror image and is endemic wherever prevalence is a property of the deployment rather than the sensor. An arrhythmia detector trained on a cardiology ward, where atrial fibrillation is common, is deployed on a healthy-population wearable where it is rare (the wearable side is Chapter 26 territory for activity, and cardiac for Chapter 18-style calibration). Each class still looks the same, \(P(x \mid y)\) is invariant, but \(P(y)\) has collapsed toward the negative class. A classifier trained under the ward prior will over-predict the positive.

The elegant correction is Black-Box Shift Estimation (BBSE), and it is where Chapter 18 pays off directly. If \(C\) is the source confusion matrix, \(C_{ij} = P_{\text{src}}(\hat{y} = i \mid y = j)\), and \(q\) is the vector of predicted-label frequencies observed on the unlabeled target, then under label shift they are linked by \(q = C\,\pi_{\text{tgt}}\), so the target prior is recovered by a single linear solve \(\pi_{\text{tgt}} = C^{-1} q\). You then rescale each class posterior by \(\pi_{\text{tgt}}(y) / \pi_{\text{src}}(y)\) and renormalize. The method needs no target labels, only target predictions, but it leans hard on the classifier being well calibrated: a biased \(C\) yields a biased \(\pi_{\text{tgt}}\), which is why the calibrated probabilities of Chapter 18 are the prerequisite and not an afterthought.

import numpy as np

def bbse_target_prior(conf_matrix, target_pred_freq):
    """Estimate the target class prior from a source confusion matrix
    and the predicted-label frequencies observed on unlabeled target data.
    conf_matrix[i, j] = P_src(pred = i | true = j). Returns a prior over y."""
    pi = np.linalg.solve(conf_matrix, target_pred_freq)   # solve q = C @ pi
    pi = np.clip(pi, 0, None)                              # priors are non-negative
    return pi / pi.sum()                                  # renormalize to a simplex

def relabel_weights(pi_src, pi_tgt):
    """Per-class posterior rescaling factor pi_tgt / pi_src (label-shift fix)."""
    return pi_tgt / pi_src

# Source (ward): AF present in 30% of windows; target (consumer wearable): rare.
C = np.array([[0.93, 0.07],      # pred=normal  given true normal / true AF
              [0.07, 0.93]])     # pred=AF      given true normal / true AF
pi_src = np.array([0.70, 0.30])
q_target = np.array([0.955, 0.045])          # what the model predicts in the field
pi_tgt = bbse_target_prior(C, q_target)
print("estimated target prior:", pi_tgt.round(3))
print("posterior rescaling   :", relabel_weights(pi_src, pi_tgt).round(3))

Code 66.1.1 makes the asymmetry with covariate shift concrete: here we correct a low-dimensional prior over classes, not a density ratio over the whole feature space, so the estimator is stable even with modest unlabeled samples. The one guardrail is that \(C\) must be invertible and well-conditioned, which fails when classes are barely separable, a diagnosis your source-side confusion matrix already reports.

Concept shift: when the meaning of the signal moves

Concept shift is the one that breaks the comfortable machinery, because the invariant that covariate and label corrections both relied on, a fixed conditional, is precisely what has changed. The same \(x\) now maps to a different \(y\): a gesture that meant "confirm" is remapped to "cancel" after a firmware update; a "normal" gait pattern becomes "abnormal" as a patient's disease progresses; a network-traffic signature that was benign becomes an attack after an exploit is published. No re-weighting of unlabeled inputs can recover a rule that the unlabeled inputs no longer determine.

Two practical consequences follow. First, concept shift is only detectable with labels, however sparse. A stream of unlabeled data can reveal that \(P(x)\) moved, but it is blind to whether \(P(y \mid x)\) moved, so a thin, continuous trickle of ground truth (audited alarms, clinician confirmations, periodic re-annotation) is the single most valuable monitoring investment you can make. Second, concept shift is often gradual and adversarial rather than abrupt: sensor wear, seasonal drift, or an adversary probing your detector all bend the conditional slowly, which is why the running "sensor failure modes" thread and the change-detection tools of Chapter 12 recur throughout this part. When your calibrated confidence stays high while a small labeled probe shows accuracy falling, you are almost certainly looking at concept shift, and the correct response is retraining or test-time training with labels, not the label-free adaptation of the next sections.

What's Next

In Section 66.2, we sharpen a case the taxonomy quietly assumed away: inputs that belong to no training class at all. Covariate shift stretched \(P(x)\) within the known label space; out-of-distribution and novelty detection ask what to do when a window arrives that the model has no right to classify, and how a calibrated confidence score becomes the first line of defense.