Foundation-model reconstruction for anomalies

"They told me I had already seen a million machines, so surely I would recognize this one. I had never heard a bearing before in my life, but I nodded and reconstructed it anyway."

A Pretrained AI Agent

This section assumes the residual-detector contract of Section 37.4 (model normal, score the miss) and the two model families it defines. The pretrained models themselves come from Part V: the general time-series backbones of Chapter 19 and the sensor- and wearable-specific ones of Chapter 20, both trained with the masked and contrastive objectives of Chapter 17. Because a pretrained model may have touched public benchmark data during training, the leakage rules of Chapter 5 get a new twist here, and we flag it throughout.

From one model per machine to one prior for all machines

The structural shift is simple to state. A conventional detector learns \(g_m\) or \(f_m\) separately for each machine \(m\) from that machine's normal data. A foundation-model detector uses a single pretrained \(\Phi\) for every machine and lets the input context, not the weights, carry machine identity. Concretely, you feed \(\Phi\) a short window of the target machine's recent healthy behavior and ask it to fill in a masked span or forecast the next samples; the residual against the true values is the score. The per-machine training cost collapses from "fit a network on months of data" to "run a forward pass," which is exactly what you want on the day a new asset comes online with no history at all. This is the cold-start case, and it is the whole reason the approach earns a place next to the per-machine detectors that otherwise dominate: it converts commissioning from a data-collection wait into an immediate deployment.

What makes this plausible at all is transfer. A model that has reconstructed millions of diverse series has internalized generic structure, periodicity, trend, noise scale, cross-channel coupling, that recurs across domains. A fault violates that generic structure just as it violates a per-machine model's specific structure, so the residual still spikes. The bet is that a broad prior, conditioned on a little context, approximates a narrow per-machine model well enough to be useful, and does so without the machine's training curve.

Two routes to a foundation-model residual

Foundation models generate anomaly residuals in the same two flavors as Section 37.4, and the choice of pretrained model largely dictates which one you get.

The forecasting route uses a pretrained zero-shot forecaster: give it context \(x_{t-w:t}\), take its point (or median) prediction \(\hat{x}_{t:t+h}\), and score \(s_t = \lVert x_t - \hat{x}_t\rVert\). This needs nothing machine-specific because good forecasters extrapolate an unseen series from context alone. The reconstruction route uses a masked encoder: corrupt or mask patches of the window, ask the model to fill them, and score the fill error \(s_t = \lVert x_t - \Phi(\tilde{x})_t\rVert\). Masked reconstruction is the more natural fit for anomaly detection, because it is exactly the pretext task of Chapter 17 repurposed at inference time, and it looks at the whole window rather than only the causal past.

Where the pretraining gap bites

The zero-shot dream fails most often on a mismatch between what the model was pretrained on and what an industrial sensor produces. Three gaps dominate. First, frequency and scale: most public time-series corpora are low-rate business and energy series (hourly demand, daily sales), while machine-condition signals are high-rate vibration and acoustic streams whose informative content lives in spectral structure the model never saw. A forecaster that is excellent on electricity load can be worthless on a raw 20 kHz accelerometer channel, so practitioners often feed foundation models an envelope, a spectrogram row, or a downsampled feature stream (the representations of Chapter 7) rather than the raw waveform. Second, context length: capturing a slow duty cycle may need far more history than the model's window admits. Third, the reconstruction-quality paradox from Section 37.4 returns in a sharper form: a very capable model can reconstruct a subtle fault too well from surrounding healthy context, driving the residual down exactly when you need it up. Foundation models do not escape the identity-shortcut trap; their scale can deepen it.

import numpy as np
from momentfm import MOMENTPipeline   # pip install momentfm

# Load a pretrained masked time-series model in reconstruction mode.
model = MOMENTPipeline.from_pretrained(
    "AutonLab/MOMENT-1-large",
    model_kwargs={"task_name": "reconstruction"},
)
model.init(); model.eval()

def foundation_residual(window, mask=None):
    """Zero-shot reconstruction anomaly score for one (1, C, T) healthy-ish window.
    No per-machine training: the pretrained model reconstructs, we score the miss."""
    import torch
    x = torch.tensor(window, dtype=torch.float32)         # (1, C, T)
    with torch.no_grad():
        out = model(x_enc=x, input_mask=mask)             # masked reconstruction
    recon = out.reconstruction.numpy()
    return np.abs(window - recon)[0]                       # (C, T) per-point residual

# Fit the alarm threshold ONLY on a stretch known to be healthy (leakage-safe).
healthy = load_normal_window()          # (1, C, T) from commissioning, this machine
resid_ref = foundation_residual(healthy)
thr = resid_ref.mean() + 4.0 * resid_ref.std()

live = load_recent_window()             # same machine, unknown state
score = foundation_residual(live).mean(axis=0)   # aggregate channels
alarms = np.where(score > thr)[0]
print("first alarm index:", alarms[0] if len(alarms) else None)

The snippet makes the cost structure explicit. There is no training loop; the per-machine work is a threshold fit on a healthy segment, the same leakage-safe habit from Chapter 5 that Section 37.4 used, now doing all of the adaptation by itself. Note the subtle leakage risk unique to this setting: if the pretrained model was trained on a public benchmark you then evaluate on, its "zero-shot" residual is contaminated, so cold-start claims must be tested on assets the model provably never saw.

Adapting the prior without labels

Pure zero-shot is the extreme; in practice a thin layer of adaptation closes most of the pretraining gap while keeping the cold-start advantage. Three light options, in increasing cost, are worth knowing. Context engineering costs nothing at training time: feed richer or longer healthy context, or preprocess into the spectral features the model handles well, and the residual sharpens. Linear probing or a small head freezes the backbone and fits only a lightweight normalizer or one-class scorer on top of the pretrained embeddings, which needs only healthy data and a few minutes. Parameter-efficient fine-tuning (adapters, LoRA) nudges the backbone toward the target domain using the machine's own unlabeled normal stream, recovering much of a per-machine model's accuracy at a fraction of its data and compute. All three stay label-free, which preserves the one-class premise the whole chapter is built on. Whichever you pick, the residual still needs turning into a calibrated false-alarm rate rather than an arbitrary cutoff, which is precisely what the conformal methods of Chapter 18 supply, and the whole pipeline still has to survive the domain shift that Section 37.6 and the test-time adaptation of Chapter 66 treat head-on.

What's Next

In Section 37.6, we confront the problem that both per-machine and foundation-model detectors run into the moment they leave the lab: domain shift across machines and sites. A detector calibrated on one pump, one factory, or one sampling rate meets a sibling that sounds subtly different, and its normal-derived threshold starts firing on healthy behavior. We look at what actually shifts, why it wrecks a residual detector, and the domain-generalization and adaptation tactics that keep an anomaly score meaningful across a whole fleet.