"A plant historian never lies about what it recorded. It simply forgets to mention that the sensor was unplugged for a month."
A Skeptical AI Agent
The Big Picture
Every model in this part, from the prognostics of Chapter 36 to the ICS anomaly detectors of Chapter 38, inherits its ceiling from the quality of the telemetry underneath it. Industrial data is not merely noisy: it is structurally broken in ways that mimic the very faults you are trying to detect. A frozen sensor looks like a perfectly stable process. A dropped packet looks like a machine that stopped. A quietly rescaled tag looks like a step change worth an alarm. This section teaches you to name these failure modes, quantify them, and gate them before they reach a model, because in industrial sensing the difference between a false alarm and a missed bearing failure is almost always a data-quality decision made hours earlier.
You arrive at industrial data expecting the concerns of Chapter 5: schema drift, missing values, leakage across splits. Those all apply. But a rotating machine wired into a decades-old control network adds failure modes that clean-desk data engineering never anticipated. The signals arrive through the SCADA, PLC, and historian stack of Sections 35.1 and 35.2, and each hop in that chain corrupts data in its own signature way. Your job is to make those corruptions observable before a model silently learns them as physics.
The industrial failure taxonomy
Generic "missing data" thinking fails here because industrial gaps carry mechanism-specific fingerprints. Learn to recognize five that dominate plant telemetry.
Stuck-at (frozen) values. A sensor or its input card fails such that it reports the last good reading forever, or a fixed rail value. The what: a channel whose variance collapses to zero while the process is demonstrably running. The why it is dangerous: a frozen temperature tag reads as a beautifully controlled loop, so operators trust it and models treat it as ground truth. The how to catch: flag any window where the rolling standard deviation drops below the sensor's known quantization step for longer than physically plausible.
Historian compression artifacts. Plant historians like OSIsoft PI store data with swinging-door or deadband compression: a point is only written when the signal deviates from a linear interpolation by more than a configured band. This is lossy by design. The consequence is that your "1 Hz tag" may have real samples minutes apart, and naive resampling manufactures fake smoothness that hides transients. Always check the true event spacing, not the nominal rate.
Clipping and saturation. A 4 to 20 mA loop or an ADC pins at its rail when the true value exceeds range. The signal flat-tops at a suspiciously round number. Unlike a frozen sensor, the process is genuinely at an extreme, but the magnitude is censored, which quietly biases any regression trained on it.
Timestamp pathologies. Clocks drift, daylight-saving transitions duplicate an hour, and buffered edge devices flush a backlog with a single arrival timestamp. The result is out-of-order, duplicated, or bursty samples. Because so much industrial inference is time-frequency analysis (Chapter 7), a wrong timestamp corrupts every spectral feature downstream.
Unit and scaling drift. A technician recalibrates a tag, swaps a sensor for a different range, or an engineer edits a PLC scaling constant. The tag's meaning changes mid-stream with no schema event. This is the industrial analog of label drift and it is the single most common cause of a model that was fine last quarter mysteriously alarming today.
Key Insight
In consumer sensing, missing data is usually random and benign. In industrial sensing, missing and corrupt data is correlated with the events you care about. Sensors fail more often during the same thermal and vibration extremes that precede machine faults, and outages cluster around shutdowns, which are exactly the transitions a prognostics model must see. Dropping bad data is therefore never neutral: it is a biased deletion that removes your most informative samples. Impute and flag; do not silently discard.
Quantifying quality: from vibes to numbers
"The data looks bad" does not survive a design review. Turn each failure mode into a measurable statistic you can trend and threshold, the same way you would monitor the process itself.
Two families of metric do most of the work. Completeness is the fraction of expected samples actually present in a window, corrected for the true (post-compression) sampling. Validity is the fraction of present samples that pass physical range, rate-of-change, and cross-channel consistency checks. A useful composite is a per-tag, per-window quality score $$q = w_c \, C + w_v \, V + w_f \, (1 - F),$$ where \(C\) is completeness, \(V\) is validity, \(F\) is the frozen-fraction (share of the window with sub-quantization variance), and the weights \(w_c, w_v, w_f\) sum to one. A model then consumes both the feature and its \(q\); windows below a gate threshold are held back or flagged for the operator rather than scored blindly. This mirrors the uncertainty-aware decisioning of Chapter 18: a low-quality window should widen predictive intervals, not be laundered into a confident number.
Practical Example: the gearbox that never overheated
A wind-farm operator ran a temperature-based overheating alarm on a turbine gearbox. For eleven months it never fired, and the maintenance team praised the reliability of that unit. A quality audit later computed the frozen-fraction \(F\) per tag and found the gearbox oil-temperature channel had \(F \approx 0.97\): the RTD input card had failed in May and the historian had been faithfully storing the same 58.3 C ever since. The alarm could not fire because the input could not move. A single frozen-fraction check, trended weekly, would have surfaced the dead sensor in days rather than after a bearing failure. The lesson generalizes: a silent alarm is not evidence of health, it is evidence you must verify the sensor is alive.
A minimal quality gate you can run today
The checks above are simple enough to implement directly, which makes them ideal for an edge gateway or historian ingestion job. The routine below scores one tag's window against the three dominant failure modes and returns a composite \(q\) plus per-flaw diagnostics, so a downstream model can decide whether to trust it.
import numpy as np
def quality_score(x, t, fs_hz, lo, hi, quant, max_gap_s,
w=(0.4, 0.4, 0.2)):
"""Score one tag window. x: values, t: unix seconds, fs_hz: nominal rate."""
x = np.asarray(x, float); t = np.asarray(t, float)
n_expected = max(1, int((t[-1] - t[0]) * fs_hz))
completeness = min(1.0, len(x) / n_expected)
in_range = (x >= lo) & (x <= hi)
dt = np.diff(t)
no_stall = np.all(dt <= max_gap_s) if len(dt) else True
validity = in_range.mean() * (1.0 if no_stall else 0.7)
# frozen: rolling variance below the quantization floor
win = max(4, int(fs_hz * 5))
fro = np.array([x[i:i+win].std() < quant
for i in range(0, max(1, len(x) - win))])
frozen_fraction = fro.mean() if fro.size else 0.0
wc, wv, wf = w
q = wc*completeness + wv*validity + wf*(1 - frozen_fraction)
return {"q": round(q, 3), "completeness": round(completeness, 3),
"validity": round(validity, 3),
"frozen_fraction": round(frozen_fraction, 3)}
t = np.arange(0, 60, 1.0) # 60 s at 1 Hz
x = 58.3 * np.ones_like(t) # a frozen RTD
print(quality_score(x, t, 1.0, lo=-20, hi=120, quant=0.05, max_gap_s=3))
# -> {'q': 0.8, 'completeness': 1.0, 'validity': 1.0, 'frozen_fraction': 1.0}Notice what the range check alone would miss: the frozen RTD is perfectly in range, so validity is 1.0. Only the frozen-fraction term reveals the fault. That is the whole argument for a composite score rather than a single filter.
Library Shortcut
The hand-rolled routine above is roughly 25 lines and covers three checks. A dedicated data-validation library such as pandera or great_expectations expresses the same range, completeness, and freshness rules declaratively in about 6 lines of schema and adds versioned expectation suites, HTML validation reports, and CI integration for free. It handles the bookkeeping (which tag failed which check, over which window, versus which prior run) that becomes unmanageable by hand once you have thousands of tags. Write the checks yourself once to understand them; then let the library own the fleet-scale accounting.
Gating, imputation, and the leakage trap
Once you can score quality, you must decide what to do with a low score. Three policies, in increasing sophistication. Gate: hold the window back from the model and mark the output "insufficient data quality" rather than emitting a guess. Impute and flag: fill short gaps (forward-fill for slow process variables, model-based interpolation for dynamic ones) but carry the imputation mask forward as a feature so the model knows which samples are synthetic. Repair: correct known systematic errors, such as reversing a unit-scaling change once you have detected it. Never impute silently; an unflagged fill is indistinguishable from real data and teaches the model that flat interpolations are normal physics.
Key Insight
Quality gates are a classic leakage vector. If you fit imputation statistics, range bounds, or a frozen-detector threshold on the whole dataset and then split for evaluation, test-set information has bled into training exactly as warned in Chapter 5 and formalized in Chapter 65. Fit every quality-related parameter on training data only, and apply the frozen bounds forward in time. In a plant, "forward in time" is not optional: your gate will run on data that does not exist yet, so it must be causal.
The reward for disciplined gating shows up downstream. A regime-segmentation model (Section 35.4) that ignores quality will treat a communications outage as a new operating regime; one that consumes \(q\) will correctly label it "unknown." An anomaly detector (Chapter 37) that ignores quality will spend its false-alarm budget on saturated sensors instead of real faults. Data quality is not a preprocessing chore you finish and forget; it is a signal you monitor for the life of the asset, right alongside the process itself.
Exercise
Take a public turbofan or bearing run-to-failure dataset (for example C-MAPSS or the IMS bearing set). For each sensor channel, compute completeness, validity, and frozen-fraction over 60-second windows and plot each metric against time-to-failure. Do any channels degrade in quality as failure approaches? Now retrain a simple remaining-useful-life regressor twice: once dropping low-quality windows, once feeding \(q\) as an extra feature. Compare the RUL error near end-of-life and explain, in terms of the biased-deletion argument above, why dropping windows can make late-life predictions worse.
Self-Check
1. A temperature tag shows zero variance for two hours while the pump it monitors is clearly running. Which two failure modes could produce this, and what single additional check would distinguish a frozen sensor from a genuinely stable loop?
2. Why does forward-filling a gap without carrying an imputation mask risk teaching a model incorrect physics, and what does the mask let the model learn instead?
3. You fit your range bounds and frozen thresholds on the full three-year archive, then split 70/30 by time for evaluation. Name the specific leakage you have introduced and the causal fix.
What's Next
In Section 35.7, we turn from the integrity of the data to the integrity of the actions it triggers: safety-critical constraints. When a model's output can open a valve, trip a turbine, or hold a furnace at temperature, the quality gates of this section become the last line of defense before a decision with physical consequences, and we will see how fail-safe design, interlocks, and human oversight are engineered around inevitably imperfect sensing.