"I have never seen this turbofan before, but I have read a billion time series. The rising ripple in your vibration trace rhymes with a thousand deaths I already know."
A Well-Read AI Agent
Why borrow a brain trained on everything to predict one bearing's death
The custom sequence models of the previous section learn degradation from scratch, and they are excellent when you own thousands of run-to-failure trajectories. Most fleets do not. A wind farm may retire ten gearboxes a year; a hospital MRI compressor fails once a decade. Remaining useful life (RUL) is the rare regression problem where the target itself, actual failure, is deliberately made scarce by good maintenance. Foundation-model embeddings offer a way out of that scarcity. A time-series foundation model pretrained on billions of unrelated series has already learned the vocabulary of trends, seasonality, regime shifts, and rising variance. You freeze it, push your sensor window through once, and read off a dense vector that summarizes the window in a representation someone else paid to learn. A small head on top of that vector can fit RUL from a few dozen failures where a from-scratch network would overfit instantly. This section is about extracting those embeddings, probing them for RUL, and knowing when the borrowed brain helps and when its foreign accent hurts.
This section assumes you are comfortable with the RUL target and prediction horizons from Section 36.2 and the from-scratch sequence models of Section 36.4, which are the baseline this approach competes against. The foundation models themselves are built in Chapter 19 (generic time series) and Chapter 20 (sensor-native); the self-supervised pretraining that gives their embeddings meaning is developed in Chapter 17. Here we treat those models as a fixed feature extractor and worry only about the last mile: window to embedding to RUL.
What a foundation-model embedding is, for a degrading machine
An embedding is the model's internal summary of an input window, read out before the model's own forecasting or reconstruction head. Formally, an encoder \(f_\theta\) with frozen weights \(\theta\) maps a multivariate sensor window \(X \in \mathbb{R}^{C \times L}\) (with \(C\) channels and \(L\) timesteps) to a vector \(z = f_\theta(X) \in \mathbb{R}^{d}\), where \(d\) is a few hundred to a couple of thousand. Encoder-style models such as MOMENT expose a dedicated embedding mode that pools the patch tokens into one vector; forecasters such as Chronos, TimesFM, and Moirai expose their pre-head hidden states, which you mean-pool over time to the same effect. The claim that makes this useful is that \(z\) is degradation-sensitive: two windows from healthy machines land near each other, and as a machine ages, its window embeddings trace a path through the space toward a failure region. RUL prediction becomes regression on that path rather than on raw sensor values, and the heavy lifting of turning noisy 100 Hz vibration into a smooth trajectory is done by the frozen encoder.
Channel handling is the first practical fork. Univariate foundation models (Chronos, TimesFM) embed one channel at a time, so a 21-sensor turbofan yields 21 vectors you must concatenate or pool. Multivariate models (Moirai, and multivariate MOMENT variants) ingest all channels together and return one joint vector that captures cross-sensor structure, which matters for faults that show up only as a changing correlation between temperature and pressure rather than in any single trace. The representation-learning tradeoffs behind these choices are the subject of Chapter 13.
Frozen embeddings convert a data-hungry problem into a data-light one
The reason this works with few failures is a shift in where the parameters live. A from-scratch RUL network must estimate millions of weights from your handful of trajectories, so it overfits. With a frozen foundation model, those millions of weights are already fixed by pretraining and never touch your labels. The only thing your scarce failures fit is the small head, perhaps a few thousand parameters. You have moved almost the entire model out of the reach of your limited labels, which is exactly why the approach shines in the low-failure regime where deep sequence models collapse.
Probing: a light head on frozen features
The standard recipe is linear or shallow probing. Freeze the encoder, embed every training window once, cache the vectors, and fit a lightweight regressor, ridge regression, a gradient-boosted tree, or a two-layer MLP, to map \(z\) to RUL. Because the encoder is frozen, embedding is a one-pass preprocessing step, not a training loop; you pay the transformer cost exactly once per window and then iterate on cheap heads in seconds. The code below embeds C-MAPSS turbofan windows with a frozen MOMENT encoder and fits a gradient-boosted head, the whole prognostic model in a dozen lines.
import numpy as np
from momentfm import MOMENTPipeline # frozen T5-style time-series encoder
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import root_mean_squared_error
model = MOMENTPipeline.from_pretrained(
"AutonLab/MOMENT-1-large", model_kwargs={"task_name": "embedding"})
model.eval() # weights stay FROZEN, no fine-tuning
def embed(windows): # windows: (N, C, L) float32
Z = []
for x in windows: # one forward pass per window, no grad
out = model.embed(x_enc=x[None]) # -> (1, d) pooled embedding
Z.append(out.embeddings.squeeze(0).detach().numpy())
return np.stack(Z) # (N, d) cached feature matrix
Z_tr, Z_te = embed(X_train), embed(X_test) # unit-encoder is fixed across splits
head = HistGradientBoostingRegressor(max_depth=3).fit(Z_tr, y_train_rul)
rmse = root_mean_squared_error(y_test_rul, head.predict(Z_te))
Two disciplines keep this honest. First, the encoder must be embedded identically across train and test, which the frozen weights guarantee, but the head's normalization statistics must be fit on train only. Second, the window-to-trajectory split must respect unit boundaries: every window from one engine goes entirely into train or entirely into test, never both, or embeddings of adjacent windows leak the answer across the split. This is the same run-to-failure leakage trap that Chapter 5 makes central, and it bites embedding pipelines especially hard because the frozen features are smooth and highly autocorrelated in time.
The encoder is the model you did not have to write
Building a comparable representation from scratch means a patch tokenizer, a transformer encoder, a masked-reconstruction pretraining loop, and a pretraining corpus, several hundred lines and days of GPU time before you fit a single RUL label. The momentfm (or Hugging Face transformers) call collapses all of it to the two lines above: from_pretrained downloads a model already pretrained on the Time-Series Pile, and .embed() returns the pooled vector. You write only the cache-and-fit head. The library owns tokenization, the pretrained weights, pooling, and batching; you own the last dozen lines.
Frozen, fine-tuned, or hybrid: choosing the coupling
Freezing is not the only option, and the right coupling depends on how far your machine sits from the pretraining distribution. Keep the encoder frozen when failures are very scarce (tens of trajectories) or when you need many per-asset heads sharing one encoder on the edge. Switch to parameter-efficient fine-tuning (LoRA adapters or unfreezing only the top block) when you have a few hundred failures and a clear domain gap, so the encoder can bend toward your sensor's statistics without discarding its priors. Reserve full fine-tuning for the rich-data regime where you might as well have trained the sequence model of Section 36.4 directly. A useful diagnostic: embed your windows, color them by RUL, and project to two dimensions. If the frozen embeddings already separate healthy from near-failure, a linear head suffices and fine-tuning buys little; if they are tangled, the domain gap is real and adapters earn their keep.
Where the field is now
As of 2026 the strongest openly documented general-purpose time-series encoders for this use are MOMENT (Auton Lab, 2024), Chronos (Amazon, 2024), TimesFM (Google, 2024), and Moirai (Salesforce, 2024), all pretrained on broad time-series corpora and all evaluated with a frozen-embedding probing protocol. Benchmarks such as the GIFT-Eval suite and MOMENT's own prognostics track report that frozen embeddings plus a shallow head are competitive with, and in low-label regimes exceed, purpose-built RUL networks on C-MAPSS and N-CMAPSS. The open frontier is sensor-native pretraining (Chapter 20): encoders pretrained on vibration and current at machine sampling rates rather than on economic and web series, which closes the domain gap that generic models still carry into the factory.
A wind farm with nine gearbox failures
An offshore wind operator wanted RUL on gearbox bearings but had only nine run-to-failure records across a 60-turbine fleet, far too few for the LSTM their vendor proposed, which memorized all nine and predicted noise on the tenth. The reliability team instead embedded 10-minute SCADA windows (gearbox oil temperature, bearing temperature, rotor speed, power) with a frozen Chronos encoder, pooling the four per-channel vectors, and fit a gradient-boosted head with quantile loss. The frozen model had never seen a wind turbine, but its priors for "slow upward drift with rising short-term variance" transferred cleanly: the probe recovered a monotone RUL trend on held-out turbines and gave 30-to-45-day lead time on two subsequent failures, enough to schedule a vessel and a crane. When they later collected 40 more failures, a LoRA adapter on the top encoder block shaved another few days of error, the clean progression from frozen probe to light fine-tune that the data volume justified.
Pitfalls unique to embedding-based RUL
Three failure modes are specific to this approach. First, silent domain mismatch: a generic encoder pretrained on daily economic series may treat your 20 kHz vibration as pure noise and return near-constant embeddings, so always sanity-check that embeddings actually vary with age before trusting the head. Second, embedding leakage: because frozen features are temporally smooth, a random window split leaks the label through autocorrelation, inflating offline scores; split by unit and by time, as above. Third, false precision in the point estimate: a shallow head on a fixed representation gives no calibrated uncertainty by default, which is dangerous for a maintenance decision. Wrap the head in a conformal predictor to get valid RUL intervals, the machinery of Chapter 18, and hand those intervals, not a single number, to the decision economics of the next section.
Common Misconception
"A frozen foundation model needs no evaluation discipline because it never saw my labels." The encoder saw no labels, but your head did, and the smooth, autocorrelated embeddings make leakage across a naive split easier, not harder. A per-unit, time-respecting split and train-only normalization are more important here, not less, than for a from-scratch model.
Exercise: probe versus train-from-scratch under a label budget
Using the C-MAPSS FD001 subset, subsample the training engines to 5, 15, and all trajectories. (1) For each budget, fit both a from-scratch 1D-CNN RUL regressor and a frozen-embedding probe (MOMENT or Chronos plus a gradient-boosted head), and plot test RMSE versus number of training engines. (2) Identify the crossover budget below which the frozen probe wins. (3) Project the frozen embeddings to two dimensions, color by RUL, and relate what you see to whether a linear head would have sufficed. (4) Add a per-unit split check and report how much your RMSE changes when you (wrongly) switch to a random window split.
Self-check
- Explain in one sentence why frozen foundation-model embeddings outperform a from-scratch sequence model when only a dozen failures are available, referring to where the parameters live.
- Why does temporal smoothness of embeddings make a random window split more dangerous than it would be for raw-signal features?
- Give one concrete signal that tells you to move from a frozen probe to a parameter-efficient fine-tune rather than staying frozen.
What's Next
In Section 36.6, we take the RUL predictions this section produces, whether from a frozen probe or a fine-tuned encoder, and make them accountable: turning a point estimate into a calibrated uncertainty interval, then feeding that interval into the maintenance decision economics that decide when repairing early actually pays.