Masked sensor modeling

"You hid a third of the accelerometer window and asked me to guess it back. After a million windows, I stopped guessing and started understanding walking."

A Self-Supervised AI Agent

Why reconstruct what you already have?

The pretext looks circular: hide data you possess, predict it back. The value is not the prediction but the representation the network must build to make that prediction possible. To fill a masked gap in an ECG trace, a model has to encode the QRS morphology and the rhythm well enough to extrapolate the missing beat. To recover a masked axis of an inertial window, it must have learned the geometric coupling between axes that a real motion imposes. These are exactly the features a downstream classifier wants, and they were learned without a single label. Formally, masked modeling minimizes a reconstruction loss over only the masked positions,

$$\mathcal{L}_{\text{mask}} = \frac{1}{|\mathcal{M}|}\sum_{i \in \mathcal{M}} \big\lVert \hat{x}_i - x_i \big\rVert_2^2,$$

where \(\mathcal{M}\) is the set of masked patch indices and \(\hat{x}_i\) is the model's estimate of masked patch \(x_i\). Computing the loss only on \(\mathcal{M}\), not on the visible patches, is deliberate: it stops the network from taking the lazy shortcut of copying visible input through and forces every gradient to reward genuine inference about hidden content.

The geometry of the mask: time, channels, and blocks

What you hide decides what the model learns. Three masking geometries dominate, and they are not interchangeable. Random patch masking in time hides scattered temporal patches within each channel; the model learns temporal continuation and is the default for univariate or channel-independent designs. Block (span) masking hides one long contiguous stretch, which is harder than the same fraction scattered, because the model cannot lean on immediate neighbors and must reason over the full window; this better matches the real failure mode of a dropout or a sensor going dark for a second. Channel masking hides entire sensors for the window and asks the model to impute them from the surviving channels, which is only meaningful in a channel-mixing architecture (Section 15.5) and directly rehearses the missing-modality robustness that Chapter 50 depends on. Many strong recipes combine them: mask random time patches most of the time and occasionally black out a whole channel, so the encoder becomes fluent in both temporal continuation and cross-sensor imputation.

There is also an efficiency prize specific to the autoencoder design. In the masked autoencoder (MAE) recipe, the heavy encoder sees only the visible patches; the mask tokens are inserted just before a lightweight decoder. With 75 percent of patches dropped, the encoder processes a quarter of the sequence, and because attention cost grows with the square of sequence length (Section 15.4), pretraining runs several times faster and cheaper than a model that must attend over the full window.

import torch

def mask_patches(x, mask_ratio=0.5):
    # x: (batch, num_patches, patch_dim), a patchified sensor window
    B, N, D = x.shape
    keep = int(N * (1 - mask_ratio))
    # per-sample random order; smallest-noise patches are kept
    order = torch.argsort(torch.rand(B, N, device=x.device), dim=1)
    keep_idx, mask_idx = order[:, :keep], order[:, keep:]
    visible = torch.gather(x, 1, keep_idx.unsqueeze(-1).expand(-1, -1, D))
    # boolean mask marks which of the N patches were hidden (loss computed here only)
    is_masked = torch.ones(B, N, dtype=torch.bool, device=x.device)
    is_masked.scatter_(1, keep_idx, False)
    return visible, keep_idx, mask_idx, is_masked

x = torch.randn(4, 16, 32)          # 4 windows, 16 patches, 32-dim each
visible, keep_idx, mask_idx, is_masked = mask_patches(x, mask_ratio=0.5)
print(visible.shape, int(is_masked[0].sum()))   # (4, 8, 32)  8

The listing above shows why the bookkeeping matters: you must remember the original position of every kept patch so the decoder can re-slot it and add positional encoding, and you must carry is_masked so the loss is scored only where it should be.

Reconstruction targets and the smooth-signal trap

Predicting raw amplitude with a mean-squared loss is the simplest target and often enough, but it has a known weakness: because sensor signals are dominated by low frequencies, an \(L_2\) loss lets the model win by matching the slow envelope while smearing the transients that carry the diagnostic information (a foot strike, a valve knock, an arrhythmic beat). Three fixes recur. Normalize each patch to zero mean and unit variance before scoring, so the loss cannot be dominated by DC offset and gain, a trick that also stabilizes across sensors with different sensitivities (Chapter 2). Reconstruct in a time-frequency representation rather than raw amplitude, so the loss weights spectral content the way Chapter 7 does; this is the approach that works well for vibration and audio-like signals. Or add a masked frequency objective alongside the time one, which is the idea behind SimMTM and related series-modeling methods. The lesson: choose the target so that reconstructing it well is impossible without capturing the structure your downstream task cares about.

When masked modeling is the right choice

Prefer masked modeling when unlabeled sensor data vastly outnumbers labeled, when the downstream tasks are diverse enough that one reusable encoder amortizes the pretraining cost, and when your signals are continuous and locally redundant so reconstruction is a meaningful pretext. It is the natural pretraining route for the foundation models of Chapter 19. It is a weaker choice when labels are plentiful (train supervised and skip the detour), when the signal is so noisy that reconstructing it rewards memorizing noise, or when you care about invariances that reconstruction does not capture: if you want an encoder that ignores sensor placement or device, a contrastive objective that explicitly pulls augmented views together (Chapter 17) may transfer better. In practice the two are complementary, and several strong systems combine a masked reconstruction loss with a contrastive one.

What's Next

In Section 15.7, we turn from how to pretrain these transformers to how to run them cheaply once trained: quantization, patch and token reduction, distillation, and the deployment tradeoffs that decide whether a masked-pretrained encoder can actually live on a wearable or an edge gateway rather than a server.