Statistical process control

"Every process has a heartbeat. My job is to tell the arrhythmias from the times you simply breathed."

A Level-Headed AI Agent

This section assumes you are comfortable with the Gaussian, standard deviation, and tail probabilities from Chapter 4, and with the ideas of sampling rate and autocorrelation from Chapter 3. We stay with the classical charts here; residual-based thresholds get their own treatment in Section 12.3, and the sequential change-point machinery (CUSUM, BOCPD) in Section 12.4.

Common cause versus special cause

The conceptual heart of SPC, due to Walter Shewhart, is a two-way split of all variation. Common-cause variation is the intrinsic noise floor of a process operating as designed: thermal noise in the ADC, micro-vibration in a bearing, the beat-to-beat jitter of a resting heart. It is stable, predictable in aggregate, and there is no single culprit to chase. Special-cause variation is an intrusion from outside that system: a loosening bolt, a fouled electrode, a firmware regression that biased a reading. A process showing only common-cause variation is said to be in statistical control, meaning its output is a stationary random draw whose distribution you can characterize once and trust.

Why does the distinction matter so much? Because it dictates the correct response. Reacting to common-cause noise as if it were a fault, sometimes called tampering, makes things worse: you chase ghosts, adjust settings that were fine, and inflate the very variance you were trying to shrink. Ignoring a special cause lets a real fault ride. SPC is, at bottom, a formal test that runs on every new sample and answers one question: is this reading still consistent with the in-control distribution, or has a special cause arrived?

The Shewhart chart and three-sigma limits

The workhorse is the Shewhart chart. During a clean baseline period (Phase I) you estimate the in-control mean \(\mu_0\) and standard deviation \(\sigma_0\) of the monitored statistic. Then, in monitoring (Phase II), you plot each new value against a center line and a pair of control limits:

The multiplier \(L\) is almost universally set to 3, giving the famous three-sigma limits. The choice is not arbitrary. For an in-control Gaussian, the probability of a single point falling outside \(\pm 3\sigma\) by chance is about \(0.0027\), so a lone out-of-limit point is strong evidence of a special cause. Shewhart picked three because it balanced two error costs he could name without knowing the application: too tight and you drown in false alarms, too loose and you miss real shifts. That balance is quantified by the average run length (ARL), the expected number of samples until an alarm. In control, \(\text{ARL}_0 = 1/0.0027 \approx 370\): a plain three-sigma chart false-alarms roughly once every 370 samples even when nothing is wrong. Choosing \(L\) is choosing your tolerable false-alarm rate, exactly the threshold-economics tradeoff that Section 12.7 makes explicit.

Run rules and the sensitivity dial

A single point outside three sigma is a blunt instrument. It catches large, abrupt shifts but is nearly blind to a small, sustained drift, the kind a slowly degrading sensor produces. The Western Electric rules add pattern detectors that read the sequence, not just the last point. The canonical set flags: one point beyond \(3\sigma\); two of three consecutive points beyond \(2\sigma\) on the same side; four of five beyond \(1\sigma\) on the same side; and eight consecutive points on one side of the center line. Each rule targets a signature that pure limit-crossing misses, especially a mean shift too small to escape the outer band on any single sample.

Every rule you add buys sensitivity and spends false-alarm budget. Turn on all four Western Electric rules and the in-control ARL drops from about 370 toward 90, meaning you now false-alarm four times as often. There is no free lunch: the fundamental tension in every detector in this chapter is detection speed against false-alarm rate, and run rules simply let you move along that curve. For a slow drift you would rather catch early, a better tool than piling on rules is a chart with memory.

Memory charts: EWMA for small, persistent shifts

The exponentially weighted moving average (EWMA) chart replaces each raw sample with a running blend of the present and the recent past:

This is exactly the exponential smoother from Chapter 6, now pressed into service as a detector. Because \(y_t\) accumulates evidence across samples, a shift of half a sigma that no single point would flag pushes the smoothed value steadily until it crosses its (correspondingly tightened) limits. Small \(\lambda\) (say 0.1 to 0.2) gives a long memory and superb sensitivity to tiny drifts; \(\lambda \to 1\) recovers the memoryless Shewhart chart. The control limits shrink because the variance of \(y_t\) at steady state is \(\sigma_0^2\,\lambda/(2-\lambda)\), smaller than \(\sigma_0^2\):

import numpy as np

def ewma_chart(x, mu0, sigma0, lam=0.2, L=3.0):
    y, prev = np.empty(len(x)), mu0
    for t, xt in enumerate(x):
        prev = lam * xt + (1 - lam) * prev      # smoothed statistic
        y[t] = prev
    t_idx = np.arange(1, len(x) + 1)
    spread = np.sqrt(lam / (2 - lam) * (1 - (1 - lam) ** (2 * t_idx)))
    ucl, lcl = mu0 + L * sigma0 * spread, mu0 - L * sigma0 * spread
    alarms = np.where((y > ucl) | (y < lcl))[0]
    return y, ucl, lcl, alarms

rng = np.random.default_rng(0)
baseline = rng.normal(50.0, 1.0, 200)           # Phase I: in control
drift    = rng.normal(50.6, 1.0, 200)           # +0.6 sigma special cause
stream   = np.concatenate([baseline, drift])
y, ucl, lcl, alarms = ewma_chart(stream, mu0=50.0, sigma0=1.0)
print("first alarm at sample", alarms[0] if len(alarms) else "none")

The code above shows the whole mechanism: smooth, compute time-varying limits, flag crossings. Run it and the first alarm lands shortly after sample 200, catching a drift that individual points hide inside the noise.

Bringing SPC to real sensor streams

Classical SPC was designed for occasional, independent quality measurements, and two of its assumptions break on high-rate sensor data. First, autocorrelation. SPC's limits assume successive samples are independent, but a signal sampled at 1 kHz is heavily correlated from one sample to the next. Feed raw high-rate data into a Shewhart chart and the effective sample count is far smaller than the nominal one, so the naive sigma is wrong and false alarms explode. The fixes are to monitor a decimated or block-summarized statistic (a rational subgroup, one summary per second rather than per sample) or to chart the residuals of a model that has already removed the correlation, which is the bridge to Section 12.3. Second, nonstationarity: real signals have diurnal cycles, duty-cycle regimes, and thermal drift that violate the single fixed \(\mu_0\). Contextual normalization, per-regime baselines, or a slowly adapting center line keep the chart honest, connecting SPC to the streaming and online-learning machinery of Chapter 60. Get these two right and a monitor costing microseconds per sample will hold its own against models that cost a thousand times more.

What's Next

In Section 12.3, we generalize the control-chart idea beyond a single monitored value to residuals: run a model of what the signal should be, watch what it fails to explain, and set thresholds on that error stream. This is how SPC's clean limit logic survives contact with the autocorrelated, nonstationary signals that broke the naive chart, and it sets up the sequential change-point detectors that follow.