"The bearing gave out on Tuesday. The work order is dated Friday. My training labels believe the paperwork, and so, for three days, I learned that a screaming machine was perfectly healthy."
A Disillusioned AI Agent
Where industrial labels actually come from
Sensor streams are abundant and cheap; the labels that say what those streams mean are scarce, expensive, and late. In an industrial plant almost every ground-truth label a predictive model can hope for comes from one place: the maintenance record. A technician closed a work order, coded a failure, and noted a repair. That record is your only anchor tying a physical fault to a timestamp on the vibration, current, or temperature trace. This section is about that anchor and about the ways it lies. Maintenance records are delayed, batched, coded inconsistently, and often written for billing rather than for machine learning. Treat their timestamps as truth and you build a model that has quietly learned the wrong moment. Understanding label delay is the difference between a predictive-maintenance system that works and one that reports a beautiful offline number it can never reproduce on the line.
This section assumes you have the telemetry from the earlier sections of this chapter already in hand: historian tags, vibration and current signals, and the regime segmentation that tells you when a machine was actually running. Here we add the other half of a supervised dataset, the labels, and confront the single property that makes industrial labels different from the clean annotations of a vision benchmark: they arrive on their own schedule, long after the physics they describe. The leakage discipline underneath all of this is developed in full in Chapter 5, and the downstream modeling of remaining useful life picks up in Chapter 36.
What a maintenance record actually is
The system of record is usually a computerized maintenance management system (CMMS) or a broader enterprise asset management (EAM) platform: SAP PM, IBM Maximo, Fiix, and their kin. Its atomic unit is the work order: an asset identifier, a reported date, a completion date, a labor and parts cost, a free-text description, and, if you are lucky, a coded failure mode drawn from a standard such as ISO 14224 (the reliability-data taxonomy for oil and gas). Work orders come in three flavors that you must not conflate. Corrective (or reactive) orders record something that already broke. Preventive orders record scheduled service on a healthy machine. Condition-based orders record work triggered by a monitoring alarm. Only corrective orders carry a failure event, and even they mix true failures with "found nothing wrong" inspections. So the first task is never modeling; it is separating the records that describe a fault from the records that describe routine care, usually by parsing failure codes and mining the free-text field.
That free text is where most of the signal and most of the pain live. "Repl brg, high vib, ran to fail" is a genuine bearing failure with a symptom; "PM per schedule" is not. Descriptions are terse, abbreviated, misspelled, and written by different technicians across shifts and decades. Extracting a clean label from them is a text-classification problem in its own right, and it is exactly the kind of task where a language model earns its place, a thread we return to in Chapter 22 on LLM agents for operations.
Label delay: the gap between failure and its record
Here is the property that reshapes everything. Let \(t_f\) be the instant a fault physically began to matter (a crack initiated, a bearing started shedding metal) and let \(t_r\) be the timestamp the CMMS assigns to the work order. The label delay is \(\Delta = t_r - t_f\), and it is almost never zero. It is not even a fixed offset you could subtract away. It is a random, right-skewed quantity built from several independent lags stacked end to end:
An operator does not notice the symptom immediately (detection lag). The order is not written the moment it is noticed but at the end of a shift or a batch data-entry session (reporting lag). The repair is not done when reported but when a crew and a spare part are both free (scheduling lag). And the timestamp the CMMS finally stores may be the reported date, the completion date, or the data-entry date, depending on how the plant configured its fields. A delay of hours is optimistic; days to weeks is common; on low-priority assets it can be months. The consequence is stark: if you paint the sensor window right before \(t_r\) as "failure" and everything earlier as "healthy," you have mislabeled the entire genuinely-degrading interval \([t_f, t_r]\) as normal, and you have taught the model that the clearest warning signs are the definition of health.
A maintenance timestamp is a fuzzy upper bound, not a ground truth
The single most useful reframing in industrial labeling: the work-order timestamp \(t_r\) is not when the fault happened. It is an upper bound on it, corrupted by a random positive delay \(\Delta\). The true onset \(t_f\) lies somewhere in an interval ending at \(t_r\), and the width of that interval is your label uncertainty. Every downstream choice, how you window the data, where you place the decision boundary, how you score the model, should treat the label as an interval rather than a point.
From a work-order timestamp to a training label
Turning records into labels means joining two clocks: the fast sensor clock (samples at hundreds of hertz) and the slow, irregular maintenance clock (a handful of events per year). The naive join, "mark the exact sample at \(t_r\)," is both too precise and wrong. A defensible pipeline instead does three things. It aligns each work order to the nearest sensor timestamp within a tolerance, so an event logged with minute resolution does not silently drop. It expands the point into a labeling window: a pre-event window \([t_r - w,\; t_r]\) treated as the degrading class, with a margin \(m\) at the trailing edge left unlabeled to absorb the unknown delay. And it marks assets that never failed in the observation period as right-censored rather than negative, because "no failure recorded yet" is not the same as "will never fail," a distinction survival analysis in Chapter 36 makes formal. The code below builds exactly this label with pandas.
import pandas as pd
# Sensor frames (fast clock) and work orders (slow, irregular clock)
sensor = pd.DataFrame({"t": pd.to_datetime(sensor_times)}).sort_values("t")
orders = pd.DataFrame({"t_r": pd.to_datetime(order_times),
"kind": order_kinds}).sort_values("t_r")
failures = orders[orders.kind == "corrective"] # keep only true faults
W = pd.Timedelta("72h") # degrading window before the record
M = pd.Timedelta("12h") # trailing margin left UNLABELED to absorb delay
sensor["label"] = 0 # 0 = healthy, 1 = degrading, -1 = ignore
for t_r in failures.t_r:
pre = (sensor.t >= t_r - W) & (sensor.t < t_r - M) # degrading, delay-safe
gap = (sensor.t >= t_r - M) & (sensor.t <= t_r) # unknown region
sensor.loc[pre, "label"] = 1
sensor.loc[gap, "label"] = -1 # drop from training
train = sensor[sensor.label != -1] # censor the fuzzy edge
-1 and dropped, so the unknown gap between true onset and the paperwork never contaminates the training set with confidently-wrong labels.The margin M is the point of the whole exercise. It is a deliberate refusal to label the interval where you know your timestamp is untrustworthy, and it is far safer than pretending the gap is healthy. Choose W from domain knowledge of how long the failure mode takes to develop, and choose M from the observed distribution of reporting lags at your plant.
Let merge_asof do the two-clock join
The alignment step, matching every work order to the nearest sensor sample within a tolerance, is a classic asof-join. Hand-rolled with a sorted search and boundary handling it is 25 to 40 lines and easy to get wrong at the edges. pandas collapses it to one call:
aligned = pd.merge_asof(orders, sensor, left_on="t_r", right_on="t",
direction="nearest", tolerance=pd.Timedelta("5min"))merge_asof replaces a bespoke nearest-timestamp matcher: it snaps each work order to the closest sensor sample, enforces a 5-minute tolerance, and leaves unmatched orders as NaN instead of silently misaligning them.The library handles the sorted merge, the nearest-versus-backward direction, the tolerance cutoff, and the null-fill for orders with no sensor coverage, all of which are exactly the boundary cases a hand-written loop forgets.
A pump fleet where the labels were three weeks late
A water-utility reliability team built a cavitation detector for a fleet of centrifugal pumps, labeling failures from their Maximo work orders. Offline the model hit an excellent precision, then collapsed in production, flagging failures the plant insisted were healthy. The audit found the cause in the label clock, not the model. The utility batched its data entry: technicians closed a week's work orders every Friday afternoon, so the stored timestamp was systematically after the repair, sometimes by up to three weeks for lower-priority pumps. The model had learned that the post-repair, freshly-serviced vibration signature (smooth, low, distinctive) was the failure class, because that was the state of the machine at the recorded timestamp. Re-deriving \(t_f\) from the operator alarm log instead of the work-order date, and dropping the ambiguous trailing window as above, moved offline precision down a few points and moved field precision up by a lot. The honest number was lower and real.
Learning and evaluating when labels are late
Once you accept that labels are delayed and one-sided, three modeling adjustments follow. First, this is naturally a positive-unlabeled problem: recorded failures are trustworthy positives, but the absence of a record is not a reliable negative, so treat unlabeled healthy time as unlabeled rather than as confirmed-negative and use PU-aware losses or class-prior correction. Second, evaluation must be delay-aware. A prediction that fires a few hours before \(t_r\) is a success, not a false positive, because the fault was already present; scoring with a point-in-time confusion matrix punishes exactly the early warnings you wanted. Use an event-based protocol with a tolerance window, related to the change-detection metrics in Chapter 12. Third, quantify the label uncertainty rather than hiding it; conformal and calibration methods in Chapter 18 let a model express a prediction interval that reflects the fuzzy boundary the labels themselves carry.
Common Misconception
"More maintenance records will fix a noisy label set." Volume does not repair a systematic timing bias. If every timestamp in your plant is late by a delay drawn from the same skewed distribution, ten thousand mislabeled examples encode the bias more confidently than a hundred do. The remedy is not more records but a better estimate of \(t_f\) (from alarm logs, operator notes, or the sensor onset itself) and an honest margin around the part you cannot pin down.
Exercise: reconstruct the true onset
You are given six months of 1 Hz bearing-temperature and vibration data for one motor, plus three corrective work orders with only completion dates. (1) For each order, propose a procedure to estimate \(t_f\) from the sensor trace itself (hint: a trailing change-point on the vibration envelope) and compare it to the work-order date to measure the delay \(\Delta\). (2) Rebuild the labels using your estimated \(t_f\) with a delay-safe margin, and state how you would set the window width W and margin M from the delays you measured. (3) Explain why marking the never-failed months as "negative" rather than "censored" would bias a remaining-useful-life model trained later.
Self-check
- Write the label delay \(\Delta = t_r - t_f\) as a sum of named lags, and explain why \(\Delta\) is right-skewed rather than symmetric around zero.
- Why is labeling the interval \([t_f, t_r]\) as "healthy" worse than leaving it unlabeled, in terms of what the model learns?
- Give one reason "no work order on record" should be treated as right-censored rather than as a confirmed negative example.
What's Next
In Section 35.6, we widen the lens from label quality to signal quality: the missing samples, clock skew, frozen tags, unit inconsistencies, and sensor dropouts that make industrial data quality its own discipline before any model is trained.