"They pretrained me on a trillion time points and called me general. Then a technician swapped the accelerometer, and I forgot how to count steps."
A Recently Redeployed AI Agent
Prerequisites
This section assumes the time-series and sensor foundation-model recipes developed in Chapter 19 and Chapter 20, the self-supervised pretraining objectives of Chapter 17, and the sampling and units material from Part I. Distribution-shift terminology comes from Chapter 66; calibration language from Chapter 18. Appendix D lists the model hubs (Chronos, MOMENT, uni2ts, granite-tsfm) referenced throughout.
Why this section matters
Foundation models rewrote how we build systems for text and images: pretrain once on a huge corpus, adapt cheaply everywhere. The same promise is now aimed at sensor streams, and the early results are real. But the recipe does not transfer cleanly, because a sensor stream is not a document. It has units, a sampling rate, a physical origin, a device that drifts, and no web-scale corpus behind it. This section names the specific places where the LLM playbook breaks on physical signals, and turns each break into a concrete open research problem you could pick up in the capstone. The point is not that these models fail. It is that knowing precisely where they are still soft is what separates a demo from a deployment.
A large language model treats every input as a sequence of tokens drawn from a fixed vocabulary, trained on a corpus that already exists and can be revisited endlessly. A sensor stream violates every clause of that sentence at once. There is no natural vocabulary, no fixed corpus, no single sampling rate, and the generative process is physics rather than authorship (a distinction drawn in Chapter 1). The four subsections below walk through the four consequences that are still genuinely open.
What is a token when the signal is continuous?
A language model gets its vocabulary for free: words are already discrete symbols. A real-valued sensor signal has no such gift, so every time-series foundation model has to invent a tokenization, and no invention is obviously right. Chronos quantizes each scalar value into one of a few thousand bins and treats the bin index as a word, which is simple but throws away amplitude precision and couples the vocabulary to the training distribution's scale. TimesFM, MOMENT, and Moirai instead cut the series into fixed-length patches (contiguous windows of samples) and linearly embed each patch, which preserves amplitude but bakes in a patch length and, implicitly, an assumed sampling rate. Neither choice answers the physical question underneath: a patch of 32 samples spans 0.32 seconds at 100 Hz and 32 seconds at 1 Hz, so the "same" token means radically different things across deployments.
This is why a model trained on one instrument can behave strangely on another that measures the identical quantity at a different rate. The tokenization has silently encoded the sampling period. Open work here includes rate-aware and continuous-time embeddings that condition on the true sampling interval \(\Delta t\) rather than assuming it, and tokenizations that respect units so that a token carries \(\mathrm{m/s^2}\) rather than a dimensionless number. Until that is solved, the safest practice is to resample every stream to the rate the model was pretrained at, and to state that rate as loudly as you state the model name.
Checkpoint
A time-series foundation model does not consume "the signal." It consumes a tokenization of the signal, and that tokenization quietly hard-codes a sampling rate, an amplitude scale, and a channel layout. Most cross-device failures are tokenization mismatches wearing a costume.
The corpus that does not exist
Text foundation models were trained on trillions of tokens scraped from the open web. There is no open web of accelerometer traces, intracardiac electrograms, or bearing-vibration spectra. The data that exists is siloed inside hospitals, factories, and phone fleets, is expensive to label, and is legally encumbered by biometric and medical privacy rules (Chapter 34, Chapter 70). The consequence is that the scaling laws that justify the whole foundation-model bet, more data and more parameters reliably buy more capability, are simply unverified for most sensor modalities. We do not yet know the exponents, or whether the curves saturate early because sensor streams carry far less entropy per token than natural language.
So the open problems are about data efficiency rather than data volume: self-supervised objectives that extract more signal per sample (Chapter 17), synthetic and simulation-augmented pretraining that manufactures physically plausible traces (Chapter 55), and cross-modal transfer that lets a model pretrained on one biosignal bootstrap another. Whether a single "large sensor model" can absorb IMUs, ECG, radar, and vibration into one backbone, or whether physical modalities are too dissimilar to share weights profitably, is the defining empirical question of the field right now.
Research Frontier
The current open-weight frontier for general time series is Chronos (Ansari et al., 2024, "Chronos: Learning the Language of Time Series"), TimesFM (Das et al., 2024, "A decoder-only foundation model for time-series forecasting"), MOMENT (Goswami et al., 2024, "MOMENT: A Family of Open Time-series Foundation Models"), and Moirai (Woo et al., 2024, "Unified Training of Universal Time Series Forecasting Transformers"), the last built to ingest an arbitrary number of channels at mixed frequencies. On the wearable side, large biosignal encoders pretrained on tens of millions of participant-hours of photoplethysmography and accelerometry (surveyed in Chapter 30) now beat task-specific baselines on held-out cohorts. The unsettled frontier question is whether these separate efforts converge into one cross-modal physical backbone, the subject of Section 72.2, or stay a federation of modality specialists.
Heterogeneity and drift: the model goes stale in the field
Two problems that barely exist for text dominate here. The first is channel heterogeneity: deployments differ in how many sensors they carry, in what order, and at what rate, so the input shape itself is variable. A model that assumes a fixed 6-channel IMU cannot ingest a 9-channel unit, and one trained on a specific electrode montage breaks when a lead falls off. The second is non-stationarity: the very transfer function \(h\) that produced the training data drifts as hardware ages, is recalibrated, or is swapped for a different part number. A pretrained model has no way to know that today's accelerometer bias is not last month's.
The standard mitigation, borrowed by nearly every modern time-series model, is reversible instance normalization: normalize each input window by its own statistics before the backbone, then de-normalize the output. It removes the per-window shift and scale so the network sees a stationary distribution regardless of device gain. The snippet below shows the whole trick.
import numpy as np
def instance_norm(window, eps=1e-5):
"""Per-window reversible normalization (RevIN, no learnable affine)."""
mu = window.mean(axis=-1, keepdims=True)
sigma = window.std(axis=-1, keepdims=True)
normed = (window - mu) / (sigma + eps)
return normed, (mu, sigma) # keep stats to invert later
def instance_denorm(normed, stats, eps=1e-5):
mu, sigma = stats
return normed * (sigma + eps) + mu
# same physical motion seen by two devices with different gain and offset
motion = np.sin(np.linspace(0, 8*np.pi, 256))
device_a = 1.0 * motion + 0.0
device_b = 2.7 * motion + 4.5 # higher gain, DC bias
na, _ = instance_norm(device_a)
nb, _ = instance_norm(device_b)
print("max abs diff after norm:", np.abs(na - nb).max()) # ~1e-6
Run it and the two devices' normalized windows agree to within floating-point noise, which is exactly why the technique is everywhere. But read the caption's warning: instance normalization only cancels first-order shift and scale. It does not touch a change in spectral content, a new noise source, or a nonlinearity that appears as the sensor ages. Those are the drifts that silently degrade a deployed foundation model, and detecting and adapting to them online, without labels, is an open problem shared with Chapter 66 on test-time adaptation and Chapter 62 on continual learning.
A predictive-maintenance model that aged badly
A wind-farm operator deployed a pretrained vibration foundation model to flag gearbox faults across 40 turbines. For six months it was excellent. Then a maintenance cycle replaced the accelerometers on a dozen turbines with a newer part that had a slightly higher resonant frequency. No fault occurred, yet the model's anomaly score crept up across exactly those twelve units, because the pretraining distribution never contained that resonance and instance normalization could not remove a frequency-domain change. The fix was not a bigger model. It was a per-device recalibration pass plus a drift monitor on the input spectrum. The lesson generalizes: a sensor foundation model's accuracy has a shelf life set by the stability of the hardware under it, and that shelf life must be measured, not assumed.
What "zero-shot" is allowed to mean
Foundation-model papers advertise zero-shot performance, meaning the model is evaluated on a task it never trained for. For text this is well defined. For sensors it is treacherous, because the pretraining corpus and the evaluation set can share a subject, a device, or a segment of the same recording, and the "zero-shot" number then measures memorization rather than transfer. This is temporal and identity leakage at foundation scale, and it is easy to commit by accident when your pretraining data is scraped from the same fleets your benchmark draws from (Chapter 65). A defensible sensor foundation-model evaluation demands cohort-disjoint and device-disjoint splits, and reports whether any evaluation device, subject, or recording appears anywhere in pretraining.
The deeper open problem is that we lack agreed benchmarks that test the properties that actually matter for physical deployment: robustness to a swapped sensor, calibrated uncertainty under drift, and graceful behavior when a channel drops out. Forecasting-accuracy leaderboards do not measure any of these. Section 72.6 returns to this evaluation gap as a first-class research direction. For now, treat every zero-shot claim as a hypothesis to be re-tested on your own disjoint data before you trust it.
Loading a pretrained forecaster in five lines
Building a patch-transformer forecaster from scratch, with tokenizer, positional encoding, masked pretraining loop, and instance normalization, is comfortably 300 or more lines before it trains at all. A foundation-model hub collapses inference to a handful:
from chronos import ChronosPipeline
import torch
pipe = ChronosPipeline.from_pretrained("amazon/chronos-t5-small",
device_map="cpu",
torch_dtype=torch.float32)
forecast = pipe.predict(context=torch.tensor(history), prediction_length=64)
median = forecast.median(dim=1).values # probabilistic, out of the box
The library gives you the model in minutes. It does not give you a disjoint evaluation split, a drift monitor, or a resampling policy matched to the pretraining rate. Those remain your responsibility, and they are where deployments live or die.
Exercise
Take a pretrained time-series foundation model from Appendix D (Chronos or MOMENT) and one accelerometer recording. (1) Evaluate zero-shot forecasting error at the recording's native rate. (2) Resample the recording to half and double that rate and re-evaluate without changing the model. Explain the error changes in terms of the tokenization's hidden sampling-rate assumption from the first subsection. (3) Corrupt the test window with a constant gain of 3x and a DC offset, run it with and without the instance-normalization wrapper above, and report which failure instance normalization fixes and which it cannot.
Self-check
- Why does a patch-based tokenizer implicitly encode a sampling rate, and what breaks when the deployment rate differs from the pretraining rate?
- Reversible instance normalization removes device gain and offset. Name one kind of sensor drift it provably cannot remove, and say why.
- What two disjointness conditions must a sensor foundation-model split satisfy before a "zero-shot" number is trustworthy, and which leakage does each prevent?
What's Next
In Section 72.2, we take the most ambitious answer to the data and heterogeneity problems seriously: a single universal encoder that ingests every modality at once. We will ask when sharing one backbone across IMU, radar, biosignal, and audio actually helps, when it merely averages away what makes each modality informative, and what cross-modal physical AI would need to earn the word "universal."