"I was handed a week of wrist data with the watch off for two nights, on the charger every afternoon, and a lost PPG lock during every run. My predecessor asked me to impute it first. I asked instead: why pretend the gaps are not there?"
A Pragmatic AI Agent
The big picture
Section 20.1 argued that raw multi-sensor streams deserve their own foundation models rather than being flattened into univariate time series. This section studies the most concrete realization of that idea to date: Google's Large Sensor Model (LSM) and its successor LSM-2, pretrained on tens of millions of hours of consumer wearable data. Their central lesson is not a clever architecture; it is a change of attitude toward missing data. Real wearable streams are incomplete by default: the device is charging, the sensor is off to save battery, the optical lock is broken by motion, the strap is loose. Classical pipelines impute those gaps and then train as if the data were whole. LSM-2 instead makes missingness a first-class citizen of the pretraining objective through Adaptive and Inherited Masking (AIM), so the model learns directly from incomplete data and degrades gracefully when more of it goes missing at inference time.
This section assumes you know how masked self-supervised pretraining works from Chapter 17 and how a Vision-Transformer patch-and-attend backbone processes a 2D grid of tokens from Chapter 15. It builds on the time-series foundation models of Chapter 19 but generalizes them from one channel to a synchronized panel of heterogeneous sensors. The signals in play (PPG, accelerometry, electrodermal activity, skin temperature) are the wearable biosignals introduced in Chapter 30.
What makes a model a "large sensor model"
What LSM does that a univariate time-series foundation model does not is treat a window of wearable data as a single two-dimensional object: sensor channels along one axis, time along the other. A day of minute-level data from a dozen derived channels becomes a grid, tokenized into patches exactly as a Vision Transformer tokenizes an image, then reconstructed under a masked-autoencoder objective. Why this framing matters is cross-channel structure: heart rate, motion, and skin temperature co-vary, and a model that attends across channels can infer a masked heart-rate patch from the accelerometer and temperature patches beside it. How the original LSM established credibility was through scaling laws. Trained on roughly 40 million hours of multimodal data from more than one hundred thousand consenting participants, its reconstruction error followed clean power laws in data volume, parameter count, and compute, the same signature that justified scaling language models. The practical payoff was strong zero-shot and few-shot performance on generative tasks (imputation of interior gaps, interpolation between samples, and temporal extrapolation) without task-specific training.
Key insight: the panel is the token grid
Once you see a multi-sensor window as a channel-by-time image, the entire Vision-Transformer toolkit transfers: patch embeddings, positional encodings on both axes, and masked reconstruction. The move that a large sensor model makes over a time-series foundation model is not a new loss; it is a new substrate. Attention now flows across sensors as well as across time, so the prior learned during pretraining is a joint prior over how physiological channels move together, not a bag of independent per-channel priors.
Missingness is the real problem, not an afterthought
The uncomfortable fact about wearable data is that it is almost never rectangular. Why gaps dominate: batteries force duty cycling, so a sensor may sample for ten seconds a minute; charging removes the device entirely for an hour a day; motion artifacts break the optical PPG lock during exactly the exercise you most want to measure; and different sensors run at different rates, so aligning them onto one grid manufactures holes wherever a slow channel has no sample. In the corpora behind these models a large fraction of the grid cells are simply absent. A standard masked autoencoder assumes a complete input, then adds synthetic masking as its learning signal. Feed it real data and you face a dilemma: impute the missing cells first (injecting a second model's errors and biases into every downstream result) or drop any window that is not complete (discarding most of your data and biasing the survivors toward users who wear the device perfectly). Neither is acceptable at foundation-model scale. This is the same missing-modality challenge formalized for deep fusion in Chapter 50, here pushed to the point where the missingness is the majority of the signal.
Adaptive and Inherited Masking: learning from incomplete data
LSM-2's answer is to unify two kinds of masking under one mechanism. Inherited masking takes the missingness that already exists in the raw data (the device-off, sensor-off, artifact-corrupted cells) and marks those tokens as unobserved. Adaptive masking adds the artificial masking that self-supervision needs, but it adapts to what is already gone: it hides a target fraction of the genuinely observed cells rather than blindly masking a grid that is mostly holes. How the objective then works: the encoder sees only observed-and-unmasked tokens, and the decoder is asked to reconstruct only the adaptively masked targets, which are cells known to hold real values. The inherited-missing cells are never used as prediction targets, because there is no ground truth to score against, and they are never fed to the encoder as if they were real. Formally, with observation mask \(m_t \in \{0,1\}\) (one where a value truly exists) and adaptive mask \(a_t \in \{0,1\}\) (one where a real value is hidden for prediction), the loss averages only over cells that are both real and chosen:
$$\mathcal{L}_{\text{AIM}} = \frac{\sum_{t} \, m_t \, a_t \, \big(\hat{x}_t - x_t\big)^2}{\sum_{t} \, m_t \, a_t}.$$Why this beats impute-then-train is threefold. It never fabricates a target, so no imputation bias leaks into the representation. It trains the model on the exact input distribution it will meet at deployment, incomplete windows, rather than an idealized complete one. And because inherited and adaptive masks share one code path, the same forward pass that learns from a nearly complete window also learns from one that is mostly empty. The result reported for LSM-2 across roughly two dozen downstream wearable tasks (classification of health conditions, activity recognition, metabolic and demographic targets) is not only higher accuracy than the first LSM but, more tellingly, far gentler degradation as inference-time missingness rises: drop half the sensors at test time and an AIM-trained model loses a fraction of what an impute-first pipeline loses.
Research frontier: where LSM and LSM-2 sit
As of 2025, Google's LSM ("Scaling Wearable Foundation Models", Narayanswamy et al.) and LSM-2 (learning from incomplete wearable data via Adaptive and Inherited Masking) are the largest publicly described sensor foundation models trained specifically on raw consumer wearable streams, as opposed to the univariate time-series corpora behind Chapter 19 models such as MOMENT or Chronos. The open question the field is chasing: whether native-missingness pretraining, contrastive objectives, and generative masked modeling can be combined so one backbone serves both robust classification and faithful signal reconstruction. Watch this space alongside the Apple-style biosignal models of Section 20.3.
The bookkeeping below is what AIM actually requires in code: build the two masks, keep only the tokens that are both real and unmasked as targets, and never let a missing cell contaminate the loss. The function returns the target mask you multiply into the reconstruction error above.
import numpy as np
def aim_masks(observed, mask_ratio=0.75, rng=np.random.default_rng(0)):
"""Adaptive and Inherited Masking for a channel-by-time sensor grid.
observed: bool array (C, T), True where a real value exists (inherited).
Returns (encoder_input_mask, target_mask):
encoder sees observed AND not-adaptively-masked cells;
loss is scored only on target cells (observed AND adaptively masked)."""
C, T = observed.shape
adaptive = np.zeros((C, T), dtype=bool)
real_idx = np.argwhere(observed) # only mask real cells
n_target = int(round(mask_ratio * len(real_idx)))
chosen = rng.choice(len(real_idx), size=n_target, replace=False)
for c, t in real_idx[chosen]:
adaptive[c, t] = True
target_mask = observed & adaptive # real values to predict
encoder_input_mask = observed & ~adaptive # real values the model sees
return encoder_input_mask, target_mask
obs = np.random.default_rng(1).random((12, 1440)) > 0.4 # ~60% present
enc_mask, tgt_mask = aim_masks(obs, mask_ratio=0.75)
print(f"present cells: {obs.sum()}, encoder sees: {enc_mask.sum()}, "
f"targets: {tgt_mask.sum()}, missing never scored: {(~obs).sum()}")~observed) is excluded from both the encoder input and the loss targets; adaptive masking hides a fraction of the genuinely observed cells for the model to reconstruct. The returned target_mask is exactly the \(m_t a_t\) term in the loss.Library shortcut: masked-autoencoder plumbing
Hand-rolling the full pipeline (patch embedding, dual-axis positional encoding, encoder over visible tokens, lightweight decoder, and per-cell loss gating) is roughly 250 to 400 lines. A masked-autoencoder scaffold from a library such as timm or the reference MAE implementation supplies the patchify, mask-token, and gather/scatter machinery, cutting the model definition to about 40 lines. You still write the AIM masking function above yourself (about 15 lines): the inherited-versus-adaptive distinction is the sensor-specific part no vision library ships, precisely because images do not arrive with holes.
Practical example: a stress-detection wearable that keeps working with the strap loose
A digital-health team ships a smartwatch feature that flags elevated physiological stress from PPG, electrodermal activity, and skin temperature. In the lab every channel is clean; in the wild the electrodermal contact fails whenever the strap loosens, and PPG drops during typing bursts. Their first system imputed the missing electrodermal channel with a per-user mean and fed the completed grid to a classifier. Accuracy looked fine on held-out lab data and collapsed in the field, because the imputed channel was confidently wrong exactly when stress spiked. Rebuilt on an LSM-2-style backbone fine-tuned with AIM, the model treats the loose-strap window as genuinely partial: it reads whatever real channels remain and abstains from inventing the missing one. Field accuracy under realistic missingness recovers most of the lab gap, and the calibration curve stays honest because no fabricated feature is driving false alarms. The leakage-safe split that made this evaluation trustworthy follows Chapter 5: users, not windows, are partitioned across train and test.
Exercise: measure graceful degradation
Take a pretrained masked sensor encoder (or the scaffold above) and a labeled downstream task with complete windows. Simulate deployment missingness by randomly zeroing out whole channels for a fraction \(p \in \{0, 0.2, 0.4, 0.6\}\) of test windows, using the observation mask rather than imputing. Plot downstream accuracy against \(p\) for two conditions: (a) the model fine-tuned with AIM-style masking, and (b) the same model with a mean-imputation front end. Report the slope of each curve. Which one degrades more gently, and does the gap widen as \(p\) grows? Explain the result in terms of where each pipeline injects error.
Self-check
- Distinguish inherited masking from adaptive masking in one sentence each. Which one supplies the self-supervised learning signal, and which one encodes real-world data absence?
- Why are inherited-missing cells excluded from the reconstruction loss rather than assigned a target of zero? What failure would a zero target cause?
- A colleague proposes imputing all gaps with a strong per-user model, then training a standard masked autoencoder on the completed grids. Name two concrete ways this can inflate offline metrics while hurting field performance.
What's Next
In Section 20.3, we turn to the other pole of wearable foundation models: Apple-style biosignal encoders that pretrain PPG, ECG, and accelerometer streams with self-supervision at population scale, and we compare their design choices (per-modality tokenization, participant-level contrastive objectives, on-device deployment constraints) against the unified masked-grid approach LSM and LSM-2 take here.