Motion/HAR foundation models

"Everyone has an accelerometer and almost no one has labels. The whole trick is turning the first fact into a cure for the second."

A Resourceful AI Agent

This section builds directly on Chapter 17 for the self-supervised pretext tasks and contrastive losses we reuse here, and on Chapter 23 for the physics of accelerometers and gyroscopes. The downstream task itself (windowing, label taxonomies, subject-wise evaluation) is the subject of Chapter 26; here we focus on the pretrained encoder that feeds it. Sections 20.2 and 20.3 covered general large sensor models and biosignal-centric wearable models; motion earns its own section because its failure modes and its supervision sources are distinct.

Why motion needs a dedicated foundation model

An inertial measurement unit (IMU) is a deceptively hostile sensor for transfer learning. The same physical activity produces wildly different signals depending on four nuisance factors that a generic time-series model does not know to be invariant to. On-body placement: a walk logged at the wrist, the hip, and the ankle looks like three different signals. Orientation: rotate the device on the strap and the gravity vector redistributes across axes, so raw \(x, y, z\) channels permute and flip. Sampling rate and range: research rigs log at 100 Hz, consumer watches downsample to 25 or 30 Hz to save battery, and full-scale ranges differ. Population and behavior: gait, cadence, and activity repertoire vary by age, health, and culture. A model trained supervised on one lab dataset silently overfits all four, which is why a classifier that scores 95% within a dataset can collapse below chance on a new cohort.

A motion foundation model is worth the trouble precisely when it bakes invariance to these nuisances into the representation rather than the classifier head. That is achieved two ways, treated in the next two subsections: pretrain on so much unlabeled data that placement and population variation are simply in the training distribution, or explicitly construct pretraining pairs that share activity content but differ in the nuisance, and force the encoder to match them.

Self-supervision at population scale

The landmark demonstration that scale alone helps HAR is the UK Biobank accelerometer work of Yuan and colleagues (npj Digital Medicine, 2024), which pretrained a ResNet-style encoder on roughly 700,000 person-days of wrist accelerometry using three label-free pretext tasks: predict whether a window has been time-reversed (arrow of time), whether its channels have been permuted, and whether a segment has been time-warped. Each pretext forces the encoder to learn what real human motion looks like without a single activity label. The released harnet checkpoints from the same group (trained via the Capture-24 free-living dataset for evaluation) then fine-tune to new HAR datasets with a small labeled set and beat from-scratch training by a wide margin, especially in the low-label regime where a fresh model has nothing to stand on.

A complementary, lighter line is LIMU-BERT (Xu et al., SenSys 2021), which ports masked-token pretraining to IMU windows: mask spans of the normalized signal, reconstruct them, and keep the model tiny enough for a phone. The lesson across both is consistent with Chapter 17: reconstruction and temporal-order pretexts learn representations that transfer across users far better than any hand-crafted feature bank from Chapter 8, provided you evaluate with subject-disjoint splits so a person never appears in both train and test.

import torch, numpy as np

# Oxford self-supervised wearable encoder, pretrained on ~700k person-days,
# loaded straight from torch.hub (repo: OxWearables/ssl-wearables).
repo = "OxWearables/ssl-wearables"
encoder = torch.hub.load(repo, "harnet10", class_num=5, pretrained=True)
encoder.eval()

# One 10 s window of tri-axial accelerometer at 30 Hz -> (batch, channels, time)
window = torch.tensor(np.load("wrist_walk_30hz.npy"), dtype=torch.float32)
window = window.reshape(1, 3, 300)              # (1, xyz, 10 s * 30 Hz)

with torch.no_grad():
    feat = encoder.feature_extractor(window)    # frozen pretrained embedding
    logits = encoder(window)                    # 5-class head, ready to fine-tune

print("embedding dim:", feat.flatten().shape[0])
print("class logits  :", logits.softmax(-1).squeeze().tolist())

The snippet above turns "train an HAR model" into a download plus a small fine-tune, exactly the collapse we want. Freeze feature_extractor and train only a linear probe when you have tens of labels per class; unfreeze the top blocks when you have thousands.

Virtual IMU: cross-modal supervision from mocap and video

Population-scale self-supervision still learns from raw IMU that carries no activity label. A second, faster-growing strategy manufactures the labels by borrowing from richer modalities. Motion-capture archives such as AMASS contain thousands of labeled human motions as full skeletons; placing a virtual sensor at any joint and differentiating its trajectory twice yields synthetic accelerometer and gyroscope streams for any body location, at any sampling rate, in any orientation, complete with the activity label the mocap already carries. This directly manufactures invariance to the placement and orientation nuisances of the first subsection, because you can render the same motion at ten body sites on demand.

Three systems mark the progression. IMU2CLIP (Meta) aligns real IMU windows with the video and text of egocentric recordings using a contrastive loss, borrowing CLIP-style multimodal alignment so the IMU encoder inherits semantic structure from vision and language. IMUGPT (Leng et al., 2024) uses a language model to expand a short activity name into diverse motion descriptions, drives a text-to-motion generator, and renders virtual IMU from the result, scaling synthetic training data without any wearer. UniMTS (Zhang et al., NeurIPS 2024) unifies these ideas: it pretrains a motion encoder against text descriptions using IMU synthesized from mocap across many body locations, and reaches strong zero-shot HAR, classifying activities it never saw a real labeled example of by matching the motion embedding to the text embedding of candidate activity names. This is the same sensor-language alignment that Chapter 21 generalizes across modalities.

Engineering placement and orientation invariance

Whether you pretrain or download, the encoder must survive the nuisance factors at inference. Three practical levers do most of the work. First, orientation-robust inputs: instead of raw \(x, y, z\), feed rotation-reduced features such as the accelerometer magnitude \(\lVert a \rVert = \sqrt{a_x^2 + a_y^2 + a_z^2}\), which is invariant to how the device sits on the strap, alongside the raw axes so the model can still use direction when it helps. The orientation-estimation machinery of Chapter 24 can rotate signals into a gravity-aligned frame before the encoder sees them. Second, rotation and axis-permutation augmentation during fine-tuning, so the head learns to ignore mounting differences. Third, resampling to a canonical rate (commonly 30 Hz) so a watch and a research logger present the encoder with the same time base; multi-rate handling in general is the subject of Section 20.5. Skipping these steps is the most common reason a motion FM that looked excellent in a notebook fails on a partner's device fleet.

What's Next

In Section 20.5, we confront the messy reality that made all of this hard in the first place: real wearable data arrives with gaps, at mismatched sampling rates, and from a shifting set of devices on the same body. We cover how a pretrained sensor foundation model ingests missing, multi-rate, and multi-device inputs without silently degrading, connecting the masking strategies of the large sensor models to the practical plumbing of a live wearable stream.