"Half my inputs were absent, the other half arrived at rates that never lined up, and no two watches agreed on what 'now' meant. Somehow they still expected one embedding."
A Well-Aligned AI Agent
Prerequisites
This section builds on the sampling, timestamping, and clock-alignment machinery of Chapter 3, the missing-data taxonomy from the estimation primer in Chapter 4, and the patch-token transformer mechanics of Chapter 15. The masked-pretraining view of missingness from Section 20.2 is assumed. Familiarity with an attention mask and additive positional embeddings is enough to read the code.
The Big Picture
The tidy tensor that most models expect, a fixed set of channels, all sampled at one rate, all present, all clock-aligned, does not exist in the wild. A wrist wearable drops photoplethysmography during vigorous motion, logs accelerometer at 50 Hz and skin temperature once a minute, and runs beside a chest strap that reports ECG at 250 Hz on a phone whose Bluetooth timestamps drift. A sensor foundation model earns its keep precisely by ingesting that mess without a bespoke preprocessing pipeline per deployment. The trick is to stop treating missingness, rate, and device identity as defects to be scrubbed before the model, and start treating them as first-class information carried alongside the signal into the model. This section shows how: represent presence with an explicit mask, tokenize each stream at its own rate with a rate-aware position encoding, and describe each channel with a device and sensor descriptor so a single architecture accepts any subset of any fleet's sensors.
Missingness is the default state, not the exception
What. Sensor data is riddled with gaps: transient dropouts (a loose electrode, motion artifact), whole-channel absence (a device without a temperature sensor), off-body periods, low-battery downsampling, and packet loss over a radio link. Following the taxonomy of Chapter 4, gaps are rarely missing-completely-at-random; PPG vanishes because the wearer is exercising, so the very fact of absence is predictive of the label you care about.
Why a mask beats imputation-then-forget. The classical reflex is to impute (forward-fill, interpolate, or model-based fill) and hand a dense array downstream. That destroys information twice: it fabricates values the model cannot distinguish from real ones, and it discards the absence pattern that was itself a signal. The foundation-model stance is to carry a binary presence mask \(m_t \in \{0,1\}\) as a parallel channel, so the model sees both the (possibly placeholder) value and the fact that it was missing. Concretely, a missing patch is replaced by a learnable mask embedding, and attention is told to weight it by its presence. This is the same mechanism as masked pretraining: the model was already trained to reason about which tokens are absent, so inference-time missingness is not a distribution shift but the regime it grew up in.
Key Insight
Absence is a feature, not a hole to be filled. The pattern of what is missing, when, and why (PPG gone during motion, ECG gone off-body) often correlates with the target more strongly than the surviving samples do. A model handed a smoothly imputed array is denied that signal; a model handed value-plus-mask keeps it. The design goal is not to make missingness invisible but to make it legible.
Multi-rate inputs: tokenize each stream at its native cadence
What breaks under resampling. The naive fix for heterogeneous rates is to resample everything to a common grid, upsampling temperature to 250 Hz or downsampling ECG to 50 Hz. Upsampling wastes compute and invents high-frequency content; downsampling aliases away the QRS morphology that made ECG worth collecting (revisit the aliasing warning of Chapter 3). A rate mismatch of 250:1 between ECG and temperature makes a single grid absurd.
How to keep native rates. The modern recipe patches each stream independently at a patch length matched to its own physics, then labels each resulting token with the absolute time of its window rather than an integer position. If channel \(c\) has sampling interval \(\Delta_c\) and patch length \(P_c\) samples, patch \(k\) covers real time \(t = k \, P_c \, \Delta_c\), and that continuous \(t\) drives the positional encoding. A common construction is the sinusoidal map over real time,
$$\mathrm{PE}(t)_{2i} = \sin\!\left(\frac{t}{10000^{\,2i/D}}\right), \qquad \mathrm{PE}(t)_{2i+1} = \cos\!\left(\frac{t}{10000^{\,2i/D}}\right),$$so tokens from a 250 Hz stream and a once-per-minute stream land in a shared temporal coordinate system and attend to one another by when they occurred, not by index. Each token also carries its rate as a learned embedding, letting the model know a temperature patch summarizes sixty seconds while an ECG patch summarizes fractions of one. This continuous-time tokenization is what lets one transformer body fuse streams that share no common grid.
In Practice: an ICU multi-parameter monitor
A bedside monitor in a cardiac step-down unit streams ECG at 250 Hz, pulse oximetry at 1 Hz, non-invasive blood pressure every 15 minutes (irregular, triggered by cuff inflation), and manual nursing observations at no fixed cadence. Resampling to any common grid either drowns the ECG or fabricates a blood-pressure trace between cuffs. Instead the team patches each channel at its own rate, stamps every token with its wall-clock time and a channel descriptor, and marks the long inter-cuff gaps with the presence mask rather than interpolating. When a lead falls off during patient repositioning, the ECG tokens for that window become masked rather than missing rows, and the pooled patient state degrades gracefully instead of the pipeline crashing. This value-plus-mask, native-rate discipline mirrors the missing-modality robustness developed for fusion in Chapter 50.
Multi-device inputs: variable channels described, not hard-coded
The problem. A fleet is heterogeneous. One user has a watch with PPG and accelerometer; another adds a chest ECG strap; a third wears a ring reporting only temperature and heart rate. Placement varies (wrist versus chest versus finger), firmware versions differ, and channel counts change between recordings and even mid-session. A model with a fixed input width and a fixed channel order cannot serve this fleet without a brittle adapter per device.
The channel-as-token solution. Treat every channel as an independent token stream and describe it rather than assuming it. Each token gets an additive descriptor embedding built from metadata: sensor modality (PPG, ECG, accelerometer-x), body placement, and native rate. The transformer then accepts any subset of any order; a missing device is simply a set of tokens that were never emitted, indistinguishable in mechanism from a masked dropout. Because attention is permutation-equivariant over tokens, channel order carries no meaning and need not be standardized. This is the same modality-agnostic principle behind large sensor models that learn from incomplete data (Section 20.2): the model conditions on what each channel is, so adding a new sensor type at deployment means adding a descriptor, not retraining the input layer.
The Leakage and Alignment Trap
Multi-device pipelines invite two silent failures. First, when you fuse streams by wall-clock time, unsynchronized device clocks (Bluetooth timestamp drift of tens of milliseconds) misalign tokens; align to a shared reference using the methods of Chapter 3 before trusting cross-channel attention. Second, the presence mask and per-channel normalization statistics must be fit on the training split and by subject or device, never across the pooled dataset; a normalizer that has seen test-device statistics leaks, exactly as the split discipline of Chapter 5 warns.
A unified recipe: mask, rate, and descriptor in one tokenizer
The three problems collapse into one representation. Every token is a triple of (value patch, presence bit, descriptor), where the descriptor encodes both continuous time and channel metadata. The snippet below builds that token set for a heterogeneous, partially missing, multi-rate batch and pools it with a mask-aware attention that ignores absent tokens. The mask does double duty: it zeroes attention onto missing patches and it drives the learnable mask embedding that stands in for their values.
import torch, torch.nn as nn
class HeteroSensorTokenizer(nn.Module):
def __init__(self, d=256, n_modalities=8):
super().__init__()
self.value_proj = nn.LazyLinear(d) # patch -> token, any patch length
self.mask_token = nn.Parameter(torch.randn(d)) # stands in for absent patches
self.modality_emb = nn.Embedding(n_modalities, d) # PPG, ECG, accel, temp, ...
def time_pe(self, t, d): # continuous-time sinusoidal PE
i = torch.arange(d // 2, device=t.device)
freq = 1.0 / (10000 ** (2 * i / d))
ang = t[..., None] * freq # (..., d/2), t in real seconds
return torch.cat([ang.sin(), ang.cos()], dim=-1)
def forward(self, patches, present, t_sec, modality):
# patches:(B,T,P) values | present:(B,T) 1=observed | t_sec:(B,T) wall-clock
tok = self.value_proj(patches) # (B,T,d)
miss = ~present.bool()
tok = torch.where(miss[..., None], self.mask_token, tok) # inject mask token
tok = tok + self.time_pe(t_sec, tok.size(-1)) + self.modality_emb(modality)
return tok, present.bool() # present -> attention key mask
def masked_mean(tok, present): # pool, ignoring absent tokens
w = present.float()[..., None]
return (tok * w).sum(1) / w.sum(1).clamp(min=1e-6)
LazyLinear adapts to any patch size), absent patches are replaced by a learned mask_token, and every token carries continuous-time positional encoding plus a modality descriptor. The returned present mask feeds a transformer's key-padding mask; masked_mean pools only observed tokens so a missing device never dilutes the patient state.Passing present as a transformer's src_key_padding_mask makes attention skip absent tokens entirely, so a batch mixing a two-sensor ring with a five-sensor watch runs in one forward pass. Nothing in the body of the model knows or cares how many channels arrived.
The Right Tool
Hand-rolling mask-aware attention (building key-padding masks, guarding the masked-mean against an all-absent window, keeping missing tokens out of the softmax) is roughly 40 to 70 lines that are easy to get subtly wrong: a single unmasked padding token quietly poisons the pooled vector. PyTorch's nn.TransformerEncoderLayer accepts src_key_padding_mask directly, so the entire missing-token handling reduces to passing the presence mask you already computed, and nn.LazyLinear removes the per-rate patch-length bookkeeping. The library owns the softmax masking and shape inference; you own only the two decisions it cannot make, how to patch each rate and what metadata to put in the descriptor.
Research Frontier
How best to condition on missingness at scale is unsettled. Google's LSM-2 (Section 20.2) advances adaptive and inherited masking, learning directly from incomplete wearable data rather than requiring imputed inputs, and reports that models trained to expect missingness degrade far more gracefully under sensor dropout than models trained on clean data. Open questions span all three axes: whether continuous-time attention or explicit resampling wins for extreme rate ratios, how to build a descriptor space that generalizes to sensor types unseen in pretraining (true zero-shot device transfer), and how to propagate calibrated uncertainty when a large fraction of channels is absent, a problem the conformal methods of Chapter 18 can wrap around a mask-aware encoder's outputs.
Exercise
Take a wearable dataset with at least three channels at different rates (for example accelerometer, PPG, and skin temperature). (1) Build the value-plus-mask representation and train a small mask-aware transformer probe; record accuracy on a held-out subject split. (2) Ablate the mask: forward-fill every gap so the model cannot see absence, and quantify the drop, especially on windows where missingness correlates with the label. (3) Replace continuous-time positional encoding with a single common-grid resampling and compare, noting compute cost and any aliasing of the fastest channel. (4) At inference, drop one entire channel (simulate a device without that sensor) and report how gracefully the masked model degrades versus the imputed baseline.
Self-Check
1. Why is imputing then discarding the presence mask worse than carrying value-plus-mask into the model, in terms of the information the model can use? Give a concrete wearable example where absence itself is predictive.
2. A colleague resamples ECG (250 Hz) and temperature (1/min) to a common 10 Hz grid before the model. Name the two distinct failures this causes, one for each channel.
3. How does treating each channel as a described token let one trained model serve a two-sensor ring and a five-sensor watch without any per-device adapter, and what makes channel order irrelevant?
What's Next
A pretrained sensor model that ingests any fleet's ragged, multi-rate, partially-missing streams is still a generalist; it does not yet know this wearer's resting heart rate, gait, or skin tone. In Section 20.6, we turn to personalization and adaptation: how to specialize a pretrained sensor foundation model to an individual or a device with few labels, from lightweight per-user calibration to parameter-efficient fine-tuning, without forfeiting the cross-user generalization that pretraining bought.