The 2023-2026 shift: from bespoke pipelines to foundation models and agentic sensing

"For years I built a brand new me for every sensor. Then someone pretrained one me for all of them, and I felt relieved and just a little redundant."

A Recently Fine-Tuned AI Agent

Between 2023 and 2026 the default recipe for building a sensor system turned inside out, and yet not a single equation of the underlying physics moved. A signal \(x(t)\), a hidden state \(s\) estimated as \(\hat{s}\), a measurement model \(x = h(s) + \eta\), and a chain of stages that turns raw transduction into a decision: all of it still holds in 2026 exactly as before. What changes is the labor split inside the chain, specifically who learns the representation \(z\) and how much labeled data that learning costs. This section assumes only the notation from Section 1.3 and the constraint web from Section 1.6. It stays deliberately shallow on mechanism, because the depth lives in Part V; its job is to hand you a map, not the terrain.

From a new model per sensor to one reusable backbone

What the old default was: a bespoke pipeline, hand-built end-to-end for a single sensor and a single task. You owned the feature extractor, the labeled corpus, and the classifier, and each was welded to the sensor that produced it. Why that was expensive: the labels were the bottleneck. A bearing-fault detector needed thousands of annotated failure windows that only exist after machines have failed; a sleep-stage classifier needed nights of expert-scored polysomnography, the overnight multi-sensor sleep study a technician hand-labels into sleep stages. The representation \(z\) was relearned from scratch every time, so the cost of a new sensor was a full new project, not an increment.

The stakes here are concrete: the difference between a medical wearable that ships this quarter and one that waits a year for its labeling budget often comes down to whether it starts from a pretrained backbone or from scratch. How the foundation-model recipe replaces it: pretrain one large backbone on vast unlabeled sensor or time-series data, then adapt it to each downstream task by fine-tuning a small head or, increasingly, by prompting it zero-shot (unpacked in the next paragraph). The backbone learns a general-purpose \(z\) that already encodes seasonality (repeating cyclic patterns), transients, and cross-channel structure; your task supplies only the last mile. Time-series foundation models such as TimesFM, Chronos, MOMENT, and Moirai now forecast series they were never trained on, and wearable-scale sensor models pretrained on millions of device-hours have been shown to transfer across activity, cardiac, and metabolic tasks. When this pays off: whenever labels are scarce and unlabeled streams are abundant, which describes most real sensor deployments. Figure 1 sketches the inversion.

Zero-shot here means running the pretrained backbone on a task or series it never saw in training, with no labeled examples and no fitting step: you pass the raw context and read off a prediction, which matters because it turns a new deployment into an API call rather than a full project. It works because the model conditions on the input window and generalizes from structure it already absorbed during pretraining, so adaptation happens inside a single forward pass instead of a training loop. Reach for it when you have no labels at all or need an instant baseline, and switch to fine-tuning a small head when you can afford a modest labeled set and want the extra task-specific accuracy. In short: pretraining does not make labels obsolete, it moves the expensive learning upstream so your handful of labels only has to point, not teach.

Self-supervised pretraining: turning unlabeled streams into fuel

What makes the backbone possible is self-supervised pretraining: learning from the structure of the signal itself, with no human labels. Why it fits sensing so well is a matter of supply. Labeled sensor events \(e\) are rare and costly, but raw streams \(x[n]\) are effectively free and endless: every wearable, vehicle, and machine generates them continuously. Self-supervision converts that surplus into training signal by inventing a task the data can grade on its own, most commonly masking part of a window and asking the model to reconstruct it, or contrasting augmented views of the same window so nearby moments map to nearby \(z\). How this yields a useful \(z\): to fill in a masked slice of an inertial measurement unit (IMU) trace the model must internalize gait periodicity, sensor axis coupling, and typical transients, and those same regularities are exactly what a downstream activity or fault head needs. When it earns its keep is precisely the scarce-label regime that dominates real deployments; the mechanics, objectives, and pitfalls are developed in Chapter 17, Self-Supervised and Contrastive Sensor Learning.

import numpy as np
from chronos import ChronosPipeline   # a pretrained time-series foundation model

pipe = ChronosPipeline.from_pretrained("amazon/chronos-t5-small")
context = np.load("vibration_window.npy")            # raw sensor history, no labels
forecast = pipe.predict(context, prediction_length=64)  # zero-shot, no training

Agentic and language-interfaced sensing

A validated forecast or a fired detection is still only a number on a screen until something decides what to do with it. The second half of the shift sits on top of perception rather than inside it. What is new is that a language model can now take sensor context as input, reason about it in words, and choose an action \(a\): read the last hour of telemetry, decide a pump is trending toward a bearing fault, and open a maintenance ticket or throttle the line. In embodied systems the same idea appears as vision-language-action models, which map a camera stream and a natural-language goal directly to motor commands. Why this is a genuine change and not just a chatbot bolted on: the interface to a sensing system becomes language and goals rather than fixed dashboards and hand-coded thresholds. An operator can now ask "why did throughput drop on line 3 overnight," and the agent pulls the relevant channels, correlates them, and answers. How it is wired: the perception stack still produces \(\hat{s}\) and its uncertainty. The agent consumes that structured output as context, plans over it, and acts, ideally with the raw measurement rigor preserved underneath. When it helps most is in operations, triage, and human-in-the-loop control where the bottleneck is interpretation and coordination, not raw detection. The architecture of these systems is the whole of Chapter 22, LLM Agents for Sensing and Operations.

What stays classical: calibration, uncertainty, and safety

This is the balance the section exists to strike. Pretraining and language interfaces are real advances, but they only change how much of the pipeline you build by hand; they leave untouched the parts of sensory AI that were never about representation learning. A foundation model still consumes \(y\), the conditioned observation (the cleaned version of the raw \(x\) from the measurement model above), and \(y\) is only as trustworthy as the sensor's calibration: once \(h(\cdot)\) drifts, a bias no backbone can see enters every embedding. That physics is the subject of Chapter 2, Sensor Physics and Measurement Models, and no pretraining substitutes for it.

Uncertainty is the second survivor. A zero-shot forecast or an agent's confident recommendation is worthless for a braking decision or a clinical alert unless you know how much to trust it. Large pretrained models are often poorly calibrated out of the box, fluent and wrong with equal confidence. Quantifying and correcting that, through the probability \(p(\hat{s})\) machinery of Chapter 4 and the modern calibration and conformal methods of Chapter 18, Uncertainty, Calibration, and Conformal Prediction, becomes more important as models get more persuasive, not less. And safety, the discipline of bounding what a system does when its inputs are spoofed, out of distribution, or simply wrong, is untouched by how the representation was learned; a language-interfaced actuator that can open a valve needs the guardrails of Chapter 68, Robustness, Sensor Spoofing, and Functional Safety regardless of how clever its planner is. Leakage-safe evaluation, too, is unchanged: a foundation model that saw your test distribution during pretraining will flatter itself exactly as a bespoke model would.

This is why the classical-versus-foundation-model tradeoff runs as a thread through the whole book rather than resolving into a verdict here. It is not classical or foundation; it is a foundation model doing the representation learning it is good at, wrapped in the calibration, uncertainty, and safety scaffolding that classical sensory AI has always supplied. The chapters that excite you about pretraining and the chapters that insist on rigor are describing two halves of one competent system.

What's Next

Section 1.8 assembles the pieces of Chapter 1 into the book's organizing spine: the six verbs, measure, clean, represent, infer, fuse, and deploy. Every part that follows, classical and foundation-model alike, expands one of those six moves applied to the single problem of inferring hidden physical state from imperfect measurements.