Monitoring drift and data quality

"The dashboard was all green. Every gauge read healthy, every pipeline was fresh, no batch was ever rejected. The model had been wrong for six weeks. Nobody was watching the shape of the numbers, only their presence."

A Vigilant AI Agent

Three kinds of drift, and why you monitor each differently

"Drift" is a bucket word for three distinct phenomena, and conflating them is the most common monitoring mistake. Input drift (covariate shift) is a change in the distribution of the features, \(P(x)\): the accelerometer now sees a population that walks slower, or the microphone sits in a noisier room. Prediction drift (label shift, seen through the model) is a change in the distribution of the model's outputs, \(P(\hat{y})\): the fall-detector suddenly fires twice as often. Concept drift is a change in the relationship itself, \(P(y \mid x)\): the same vibration signature that meant "healthy bearing" last year now means "early wear" because the lubricant was reformulated. These are ordered by how easily you can see them. Input and prediction drift are computable from data you already have at inference time, with no labels at all. Concept drift is invisible to any input-only monitor: the inputs can look identical while the correct answer has silently changed, which is exactly why it is dangerous and why the proxy-and-delayed-label machinery of the next section exists.

Data quality is a separate axis from drift

Before you measure whether a distribution moved, confirm the data describing it is trustworthy, because a broken pipeline mimics drift perfectly. Data-quality monitors are cheap, unglamorous, and catch most real incidents. Freshness: what is the lag between a reading's timestamp and its arrival, and is any device silent past its expected cadence? Completeness: what fraction of expected samples actually arrived per window, and is the missingness pattern random or structured (one firmware version, one geography, one time of day)? Plausibility beyond the contract: flat-line runs, impossible slew rates, a channel that has not changed a bit in an hour. Duplication and reordering: repeated batches from a retrying uploader inflate counts and skew every downstream statistic. The reason to separate these from drift is causal attribution: a "distribution shift" that is really a gateway dropping every third packet demands a networking fix, not a retrain. Always rule out a data-quality cause before you trust a drift number, because a drift detector run on corrupted data reports drift with total confidence.

Measuring distributional drift

To detect input or prediction drift you compare a fixed reference distribution (typically the training set, or a blessed recent window that matches training performance) against a rolling detection window of live data. For a single channel, three tools dominate. The Population Stability Index bins both distributions and sums a symmetric divergence over bins, $$\text{PSI} = \sum_{i=1}^{B} (p_i - q_i)\,\ln\!\frac{p_i}{q_i},$$ where \(p_i\) and \(q_i\) are the reference and current fractions in bin \(i\); the field conventions are \(\text{PSI} < 0.1\) stable, \(0.1\!-\!0.25\) moderate, \(> 0.25\) significant. The two-sample Kolmogorov-Smirnov statistic takes the maximum gap between the empirical CDFs and gives a p-value, so it is threshold-free but sensitive to sample size. The Jensen-Shannon divergence, a bounded symmetric cousin of the KL divergence from Chapter 4, behaves well when a bin empties out. Raw sensor channels are rarely the right unit to monitor, though: a \(100\,\text{Hz}\) waveform is high-dimensional and autocorrelated. Monitor the same low-dimensional features the model consumes, spectral band powers, statistical moments, or the model's own embedding (Chapters 8 and 13), and for genuinely multivariate drift reach for a kernel Maximum Mean Discrepancy test, which compares distributions in a reproducing-kernel space without binning.

import numpy as np
from scipy.stats import ks_2samp

def psi(ref, cur, bins=10, eps=1e-6):
    """Population Stability Index between a reference and current sample."""
    edges = np.quantile(ref, np.linspace(0, 1, bins + 1))   # equal-frequency bins on ref
    edges[0], edges[-1] = -np.inf, np.inf                   # catch out-of-range live values
    p, _ = np.histogram(ref, edges); q, _ = np.histogram(cur, edges)
    p = p / p.sum() + eps; q = q / q.sum() + eps            # smooth empty bins
    return float(np.sum((p - q) * np.log(p / q)))

def drift_report(ref, cur):
    ks_stat, ks_p = ks_2samp(ref, cur)
    score = psi(ref, cur)
    level = "stable" if score < 0.1 else "moderate" if score < 0.25 else "significant"
    return {"psi": round(score, 3), "level": level,
            "ks_stat": round(ks_stat, 3), "ks_p": round(ks_p, 4)}

rng = np.random.default_rng(0)
reference = rng.normal(0.0, 1.0, 5000)          # training-time feature (e.g. gait cadence)
same      = rng.normal(0.0, 1.0, 800)           # a fresh in-distribution window
shifted   = rng.normal(0.4, 1.2, 800)           # slower, more variable cohort
print("no drift:", drift_report(reference, same))
print("drift   :", drift_report(reference, shifted))

Code 69.3.1 makes the two-window comparison concrete: freeze the reference at model-blessing time, recompute the report each detection window, and alert when PSI crosses your band or the KS p-value falls below a corrected threshold. Bucketing PSI on the reference quantiles matters, it puts equal training-time mass in each bin so the score reflects where the model actually had support, and the open outer edges ensure a live value beyond anything seen in training is counted rather than silently dropped.

Making it work at fleet scale without alert fatigue

Two devices are easy; ten thousand devices with thirty features each is a hundred thousand hypothesis tests per window, and at any fixed p-value threshold a flood of them fire by pure chance. Three disciplines keep the signal usable. First, correct for multiple comparisons: control the false-discovery rate across the features-times-devices grid (Benjamini-Hochberg) rather than testing each in isolation, or the on-call rota will mute the whole system by the second week. Second, choose the right aggregation grain: fleet-wide drift, per-cohort drift (firmware, geography, hardware revision), and per-device drift answer different questions, and a shift confined to one firmware version is invisible in the fleet average but obvious in a per-cohort split, which is also your first attribution clue. Third, tune windows and reference deliberately: too short a detection window is dominated by noise and diurnal cycles; a stale reference makes normal seasonal variation look like drift. Compare a window against both the fixed training reference (has the world moved since training?) and a trailing recent window (is this change abrupt or gradual?). The output of all this is not "retrain now", it is a scoped, attributed, deduplicated alert; converting that signal into a retraining decision is the job of Section 69.5.

What's Next

In Section 69.4, we confront the hardest case a drift monitor cannot solve on its own: concept drift under delayed or absent ground truth. When the true labels for today's sensor readings arrive weeks later, or never, you cannot measure accuracy in real time, so we build proxy signals, confidence and calibration trends (Chapter 18), agreement between redundant models, and physics-consistency checks, that stand in for the accuracy you cannot yet see.