"A language model was handed a billion sentences. I was handed a billion sensors that never agreed on what a word was."
A Well-Read AI Agent
The Big Picture
A time-series foundation model is only as good as two upstream choices most readers never see: the corpus it pretrains on and the tokenizer that turns a raw waveform into model inputs. Text has a settled answer here (scrape the web, apply byte-pair encoding). Sensor data has no such luxury. Amplitudes span nanovolts to megawatts, sampling rates span millihertz to megahertz, and no universal vocabulary exists. This section explains how the leading models assemble their training data and the two dominant tokenization philosophies (continuous patching versus discrete value tokenization) so you can predict which family will behave well on your signal before you download a single checkpoint.
This section assumes you understand sampling and amplitude scaling from Chapter 3, the leakage discipline for constructing datasets from Chapter 5, and the patch-and-attend mechanics of Chapter 15. Section 19.1 defined what earns a model the "foundation" label; here we look inside its mouth.
Corpora: where a billion time points come from
Text foundation models inherit a happy accident: the internet is a pre-assembled trillion-token corpus. Time series has nothing comparable, so each model builds its own. Three strategies dominate, usually blended.
Aggregated real archives. The largest curated open collection is LOTSA (Large-scale Open Time Series Archive, roughly 27 billion observations across energy, transport, climate, web, sales, and healthcare), assembled for the Moirai family. The classic Monash forecasting archive supplies dozens of smaller heterogeneous datasets. These give diversity of shape (trends, multiple seasonalities, intermittency) that a single domain cannot.
Web-scale weak signals. TimesFM pretrains on hundreds of billions of points harvested from Google Trends and Wikipedia page views, which are cheap, high-volume, and rich in human-driven seasonality. The bet is that broad statistical coverage beats domain purity, the same bet that made large language models work.
Synthetic generation. Because real corpora underrepresent clean periodicity and controlled trend-break structure, Chronos and TimesFM inject synthetic series built from Gaussian processes, composed sinusoids, and piecewise-linear trends (Chronos calls its generator KernelSynth). Synthetic data is unlimited, license-clean, and lets designers deliberately teach behaviors that are rare in the wild.
Key Insight
Corpus diversity matters more than corpus size for zero-shot transfer. A model that has seen ten thousand nearly identical electricity meters learns one pattern very well; a model that has seen electricity, foot traffic, river flow, and synthetic sinusoids learns the shape grammar of time series, which is what lets it forecast a vibration sensor it has never encountered. This is also why corpus composition is the single biggest leakage risk in Chapter 19: if your evaluation dataset overlaps the pretraining archive, the "zero-shot" number is a lie.
Patching: the continuous tokenizer
Patching, popularized by PatchTST and used by TimesFM, Moirai, MOMENT, and the TinyTimeMixers, treats a token as a short contiguous window of real values. A series is split into non-overlapping patches of length \(P\) (typically 16 to 64 samples), each patch is instance-normalized, then mapped by a single linear layer into a \(d\)-dimensional embedding:
$$ \mathbf{z}_i = \mathbf{W}\,\tilde{\mathbf{x}}_{iP:(i+1)P} + \mathbf{b}, \qquad \tilde{\mathbf{x}} = \frac{\mathbf{x} - \mu}{\sigma + \epsilon} $$
Two properties follow. First, patching cuts the sequence length by a factor of \(P\), which is decisive for attention that scales as \(O(L^2)\): a 2048-sample window becomes 32 tokens, not 2048. Second, the representation stays continuous, so numeric resolution is preserved exactly and the model regresses real-valued outputs directly. The cost is a fixed patch length that bakes in an implicit timescale, and a linear projection that can smear a sharp transient sitting mid-patch.
Value tokenization: the discrete tokenizer
The opposite philosophy, made famous by Chronos, discards the continuous representation entirely. It scales each series (mean-scaling divides by the mean absolute value), then quantizes every single value into one of \(B\) bins, producing an integer token exactly like a word ID. The model is then a vanilla language model (Chronos reuses the T5 architecture unchanged) trained with cross-entropy over the bin vocabulary, and it forecasts by sampling token sequences, which yields a probabilistic forecast for free.
The appeal is enormous reuse: the entire mature language-model stack (tokenizer, transformer, sampling, decoding) transfers with almost no modification. The price is quantization error and a fixed dynamic range: values beyond the outermost bins clip, and fine amplitude differences within a bin vanish. Chronos-Bolt later moved to patch-based inputs and direct multi-step decoding to claw back speed and precision, a telling convergence between the two camps.
import numpy as np
def chronos_style_tokenize(x, n_bins=256, low=-15.0, high=15.0):
# 1. mean-scaling: normalize by mean absolute value (per series)
scale = np.mean(np.abs(x)) + 1e-8
x_scaled = x / scale
# 2. quantize into uniform bins over a fixed clamped range
edges = np.linspace(low, high, n_bins - 1)
tokens = np.digitize(np.clip(x_scaled, low, high), edges)
return tokens, scale
def patchify(x, patch_len=16):
# continuous tokenizer: reshape into (n_patches, patch_len), no vocabulary
n = (len(x) // patch_len) * patch_len
return x[:n].reshape(-1, patch_len)
sig = np.sin(np.linspace(0, 12 * np.pi, 256)) + 0.1 * np.random.randn(256)
tok, scale = chronos_style_tokenize(sig)
patches = patchify(sig)
print("value tokens:", tok[:8], "| unique bins used:", len(np.unique(tok)))
print("patch tensor shape:", patches.shape) # (16, 16): 16 tokens, not 256
Right Tool: don't hand-roll the tokenizer
The demo above is for intuition. In practice, autogluon.timeseries and the chronos-forecasting package expose the trained tokenizer, scaler, context handling, and sampler behind roughly three lines: ChronosPipeline.from_pretrained("amazon/chronos-bolt-base") then pipeline.predict(context, horizon). That replaces the roughly 150 lines of scaling, binning, padding, batching, and probabilistic decoding you would otherwise maintain, and it guarantees your bin edges match the checkpoint's (a mismatch silently destroys accuracy).
Choosing a tokenizer for real sensors
The decision hinges on your signal's statistics. Patching preserves amplitude precision and short context, so it suits smooth, well-scaled sensor streams (temperature, load, flow) and multivariate inputs where Moirai's "any-variate" patching shines. Value tokenization is robust to wild scale variation across a fleet because mean-scaling plus binning is aggressively normalizing, and its native probabilistic output is valuable when you need forecast intervals rather than a point estimate, which connects directly to the calibration machinery of Chapter 18. Neither handles heavy-tailed spikes gracefully: patching averages them away, quantization clips them, so pre-conditioning (log or arcsinh transforms) is often worth more than the model choice itself.
Practical Example: vibration on a factory pump
An industrial team pointed a pretrained TimesFM patch model at a 25.6 kHz accelerometer on a centrifugal pump and got noise. The failure was not the model, it was the tokenizer's implicit timescale: a 32-sample patch spans 1.25 ms, far shorter than the 20 ms bearing-fault period, so each token saw meaningless high-frequency ripple. Downsampling the signal to an envelope at 100 Hz (so one patch covered a full mechanical cycle) turned the same checkpoint from useless to competitive. The lesson generalizes: match your sampling rate to the model's patch length before blaming the weights, a diagnostic that pairs naturally with condition-monitoring practice in Chapter 37.
Research Frontier
Tokenization is unsettled and actively contested as of 2026. Value quantization (Chronos), continuous patching (TimesFM, Moirai, MOMENT), and lag-feature tokenization (Lag-Llama) coexist with no clear winner, and Chronos-Bolt's shift toward patched inputs plus mixture-of-experts routing (Moirai-MoE, Time-MoE) that lets the model pick specialized experts per token suggests the field is converging on hybrid, resolution-aware schemes rather than one universal vocabulary. Watch for learned, frequency-conditioned patching that adapts \(P\) to the detected dominant period instead of fixing it in advance.
Exercise
Take a 10-minute PPG recording sampled at 64 Hz. (a) Compute how many patches a length-32 patcher produces and what wall-clock duration one patch spans. (b) Run the chronos_style_tokenize function above and report how many of the 256 bins are actually used; explain what that says about the signal's dynamic range. (c) Add a single artifact spike 50x the signal amplitude and describe, without running it, how each tokenizer distorts that sample.
Self-Check
1. Why does corpus diversity threaten the honesty of a "zero-shot" claim, and how does that connect to leakage-safe dataset construction?
2. State one concrete advantage and one concrete disadvantage of value tokenization versus patching.
3. A colleague reports terrible zero-shot accuracy on a 40 kHz ultrasonic sensor. What single tokenizer-related question do you ask first?
What's Next
In Section 19.3, we tour the model families themselves (TimesFM, Chronos and Chronos-Bolt, Moirai and Moirai-MoE, MOMENT, Time-MoE, the TinyTimeMixers, Lag-Llama, Toto, UniTS, and Timer) and show how each one's corpus and tokenization choices from this section directly shape what it does well and where it breaks.