"You trained me on electricity demand, river gauges, and factory vibration, then pointed me at a heart rate I had never seen and asked for the next hour. I did not ask which dataset it came from. That was rather the point."
A Transferable AI Agent
Prerequisites
This section assumes the transformer machinery of Chapter 15 and the self-supervised pretraining objectives of Chapter 17, since a foundation model is what you get when those two ideas are pushed to scale. You should be comfortable with the leakage-safe, subject-wise and time-wise splitting discipline of Chapter 5, because zero-shot claims are exactly where leakage hides. Basic notions of stationarity and sampling rate from Chapter 3 and predictive uncertainty from Chapter 4 are used lightly. This is the definitional section: it fixes what the word "foundation" means for time series so the corpora, families, and tasks in the rest of the chapter have a target to hit.
The Big Picture
The phrase "foundation model" was coined for large language and vision systems: a single model, pretrained once on broad data at scale, that adapts to many downstream tasks with little or no task-specific training. Porting that idea to time series is not cosmetic relabeling. A forecaster that is excellent on one factory line but must be retrained for the next line is a good deep model, not a foundation model. This section pins down the four properties that actually earn the name, then confronts the hard truth that time series make the port genuinely difficult: there is no shared vocabulary, no fixed scale, no natural alignment across domains, and the signal is non-stationary by default. Understanding those obstacles is what makes the design choices of the rest of the chapter, from tokenization to evaluation, feel inevitable rather than arbitrary.
The definition, ported from language to signals
The standard definition, following the framing that named the category (Bommasani and colleagues, 2021), has three moving parts: broad pretraining, at scale, yielding a single reusable model that adapts to many tasks. A large language model satisfies all three because text has a property sensor streams do not share: a universal, discrete, self-supervising target. Every document is a sequence over the same fixed vocabulary, and the next token is free supervision harvested from data that already exists. Pretrain on enough of it and the model generalizes to translation, summarization, and code without a task head per job. The same recipe built the monocular-depth foundation models of Chapter 41, where one network estimates depth on scenes it never trained on.
For time series the ambition is the same. Pretrain one model on a broad, heterogeneous collection of numeric sequences drawn from many domains, at a scale of billions of observations, so that when a new sensor stream arrives the model forecasts, imputes, or scores it with no gradient step at all. That last clause, no gradient step, is the operational heart of the definition. A task-specific model is a function you fit to a dataset; a foundation model is a function you apply to a dataset. When you can drop in a series the model has never seen (a new patient, a new pump, a new building) and get a calibrated forecast without touching the weights, you are holding a foundation model. When you cannot, you are holding a very good specialist.
Key Insight
The single sharpest test is the transfer boundary. Ask: does adapting to a new series require changing the weights? A task-specific model answers yes, because it must be fit to the new distribution. A foundation model answers no, because generalization across unseen series was baked in at pretraining time. Everything else (scale, architecture, tokenization) is machinery in service of moving that boundary from "retrain per dataset" to "run on anything."
Why time series resist the recipe
If the language recipe transferred cleanly, this chapter would be one page long. It does not, because time series lack every convenience that made text easy. First, there is no shared vocabulary. Text has roughly \(10^4\) to \(10^5\) discrete tokens common to all documents; a real-valued signal has an unbounded continuum with no natural symbol boundaries, which is exactly the tokenization problem that Section 19.2 is devoted to. Second, there is no shared scale. One channel is a heart rate near 70, the next is grid load near \(10^6\) watts, the next is a normalized vibration near zero. A model that has not been made scale-invariant will treat the megawatts as overwhelmingly important and ignore the heartbeat. Third, there is no shared semantics across domains: the correlation structure of electricity demand tells you almost nothing about bearing vibration, so pretraining data is far more heterogeneous than a corpus of human language, which is all, at bottom, the same code.
Fourth, and most corrosive, sensor streams are non-stationary. The statistics that hold in the pretraining window drift by deployment time, a problem developed fully in Chapter 66. Language is comparatively stationary; the grammar of English does not shift under your feet mid-sentence. A time-series foundation model must therefore generalize not only across domains it saw but across regime changes it did not. This is why the honest evaluation of these models, the subject of Section 19.6, is so fraught: a model can look zero-shot when the test series secretly overlaps its pretraining corpus.
The four properties that earn the name
Combining the definition with the obstacles gives a concrete checklist. A model deserves "foundation" status for time series when it exhibits all four of the following. One, single pretraining. It is trained once, self-supervised, on a broad multi-domain corpus, and shipped frozen; downstream use adds no per-dataset training. Two, task-agnostic adaptability. The same weights serve forecasting, imputation, classification, and anomaly scoring, the tasks catalogued in Section 19.4, rather than a separate model per task. Three, invariance to the nuisance axes. It absorbs arbitrary amplitude, offset, sampling rate, context length, forecast horizon, and channel count without breaking, because those vary wildly across sensors and are not the signal you care about. Four, favorable scaling. Quality improves predictably with more pretraining data and parameters, the property that separates a foundation model from a merely large one.
Property three is where most of the engineering lives, and one piece of it is worth building by hand because it is so central: scale invariance. Because a single model must see megawatts and millivolts in the same batch, foundation-model pipelines normalize each input window to a common statistical footprint before the network sees it, then invert that normalization on the output. This reversible instance normalization is what lets one frozen model accept a heart rate and a river gauge in the same call. The listing below implements it in its bare form.
import numpy as np
def instance_normalize(window, eps=1e-5):
"""Per-window standardization: strip scale and offset so one
frozen model can accept heterogeneous sensors. Returns the
normalized window plus the statistics needed to invert it."""
mu = window.mean()
sigma = window.std() + eps
return (window - mu) / sigma, (mu, sigma)
def denormalize(pred, stats):
mu, sigma = stats
return pred * sigma + mu
# Two sensors the model must handle with the SAME weights.
heart_rate = np.array([68., 71., 70., 72., 69.]) # bpm, ~O(70)
grid_load = np.array([1.2e6, 1.3e6, 1.25e6, 1.4e6]) # watts, ~O(1e6)
for name, w in [("heart_rate", heart_rate), ("grid_load", grid_load)]:
z, stats = instance_normalize(w)
print(f"{name:<10} normalized mean={z.mean():+.2f} std={z.std():.2f}")
print(f"{name:<10} recovered ={denormalize(z, stats)}")Listing 19.1.1 is the smallest possible demonstration of why one set of weights can serve incompatible sensors: strip scale and offset on the way in, restore them on the way out. It is also a reminder that a foundation model is a pipeline, not just a backbone; the normalization, patching, and horizon handling around the network are what make the frozen weights usable on arbitrary inputs.
The Right Tool
You almost never hand-roll Listing 19.1.1 in production. A pretrained time-series foundation model exposed through gluonts or the Hugging Face pipeline API bundles the instance normalization, the tokenization, the context windowing, and the probabilistic decoding behind one call. Loading a released checkpoint and forecasting a brand-new sensor is roughly three lines and handles all of property three internally, replacing on the order of a hundred lines of preprocessing, batching, and inverse-transform bookkeeping you would otherwise write and get subtly wrong. The specific model families and their APIs are the subject of Section 19.3; the point here is that the pipeline, not just the weights, is what ships.
In Practice: the hospital that never fine-tuned
A clinical engineering team monitoring the biosignals of Chapter 28 needed short-horizon forecasts of vital-sign trends across dozens of ward devices: heart rate, respiration, SpO2, and infusion-pump flow, every one on a different scale and cadence. The specialist route would have meant one trained model per signal type per device model, a maintenance burden that grew with every new monitor procured. Instead they dropped a pretrained time-series foundation model in front of the whole fleet with no training run at all. Each incoming window was instance-normalized as in Listing 19.1.1, forecast by the frozen model, and denormalized back to clinical units. A new pump model added to the ward the next quarter worked on day one, no retraining, because it was just another normalized sequence to a model whose transfer boundary already sat at "any numeric series." The four properties were not abstractions on a slide; they were the reason a single deployment covered signals the team had not even purchased yet.
Research Frontier
The first models to plausibly satisfy all four properties arrived in 2023 and 2024. Google's TimesFM (Das and colleagues, 2024) pretrained a decoder-only patched transformer on the order of a hundred billion time points and reported zero-shot forecasting competitive with fully supervised specialists across the Monash and other benchmarks. Amazon's Chronos (Ansari and colleagues, 2024) took the opposite tokenization stance, quantizing values into a discrete vocabulary so an off-the-shelf language-model architecture could be trained on it, and still generalized zero-shot. Salesforce's Moirai and the masked-reconstruction model MOMENT extended the recipe to any-variate inputs and to the imputation and classification tasks of Section 19.4. The live and unsettled question is whether these systems truly generalize or merely memorize an enormous, overlapping corpus; the leakage and non-stationarity concerns that the GIFT-Eval and related benchmarks of Section 19.6 exist to expose are the field's current proving ground, and the honest answer for many claimed zero-shot wins is still "measure again on data the model provably never saw."
Exercise
Take three labeled sensor datasets from different domains (for example a HAR accelerometer set from Chapter 26, an ECG trace, and a building-energy meter). (1) Using Listing 19.1.1, instance-normalize a window from each and confirm all three land at zero mean and unit variance; explain in one sentence why this is a precondition for a single frozen model to serve all three. (2) Score a model you have access to against the four-property checklist: does adapting to a new dataset change the weights, does one set of weights serve more than one task, is it invariant to scale and horizon, and does its published quality improve with scale? (3) Argue, for one of your datasets, whether a foundation model or the task-specific model of Chapter 14 is the better bet, and name which property decided it.
Self-Check
1. State the single operational test that separates a foundation model from a very good task-specific model, and explain why "no gradient step on the new series" captures it.
2. Text gave language models a universal vocabulary and free next-token supervision. Name three properties of raw time series that break this recipe, and for each say which later section of this chapter addresses it.
3. Why is reversible instance normalization (Listing 19.1.1) a prerequisite for a single frozen model to forecast both a 70 bpm heart rate and a megawatt grid load, and what would go wrong without the inverse step?
What's Next
In Section 19.2, we tackle the first obstacle head-on: with no natural vocabulary for real-valued signals, how do you turn a continuous sensor stream into tokens a transformer can pretrain on? You will compare the two dominant answers, patching (slicing the series into overlapping windows treated as tokens) against value tokenization (quantizing amplitudes into a discrete codebook), and see how that single choice shapes everything the resulting foundation model can and cannot do.