Cross-device, cross-user, cross-modal pretraining

"You trained me on your left wrist. I now meet a chest strap, a stranger, and a microphone in the same second. Tell me again which of us is the anomaly."

A Well-Traveled AI Agent

Three gaps, one geometry

Name the shifts precisely, because each demands a different move. The device gap is a change in the measurement model: a new IMU has a different sensitivity, bias, sampling rate, and axis convention, so the same motion produces a numerically different window. The user gap is a change in the underlying signal source: person B's ECG morphology, stride length, or perfusion differs from person A's, so even an identical device yields a different distribution. The modality gap is structural: accelerometer, gyroscope, PPG, audio, and barometer live in different physical units, at different rates, with different information about the same latent event.

The reason one section can cover all three is that self-supervised learning already gives you the two verbs you need. To close a gap by invariance, you make windows that differ only along the nuisance axis into a positive pair, so the encoder is trained to ignore that axis. To close a gap by alignment, you take two genuinely different signals of the same event and pull their embeddings together, exactly the cross-modal contrastive move that CLIP made famous for images and text. Device and user gaps are usually attacked by invariance; the modality gap is usually attacked by alignment, though the boundary blurs.

Closing the device and user gaps

Device invariance has three complementary levers. First, canonicalize the input so devices are less different before the network sees them: resample every stream to a common rate (Chapter 3), normalize units, and, for tri-axial inertial data, remove the orientation nuisance with rotation augmentation or a gravity-aligned frame. Second, treat device identity as an augmentation: if you have any windows recorded simultaneously on two devices, they are a free, physically exact positive pair, better than jitter or scaling because the difference is the true hardware gap rather than a guess at it. Third, discourage the encoder from encoding device identity at all, either with a domain-adversarial head that is trained to predict the device while the encoder is trained to fool it, or simply by including enough device diversity that device becomes uninformative.

The user gap yields to the same toolkit with one addition: batch and split by subject. Positives drawn across subjects who share an activity push the representation toward person-agnostic features, and, critically, evaluation must use leave-one-subject-out or grouped splits so that no window from a test person ever appears in training. Subject-level leakage is the single most common way a wearable paper reports a number it cannot reproduce in the field; the grouped-split discipline of Chapter 5 is not optional here. When some personalization is allowed at deployment, a small per-user adapter or a few labeled calibration windows recovers most of the person-specific accuracy without retraining the backbone, the pattern developed for wearable foundation models in Chapter 20.

Closing the modality gap by alignment

Cross-modal pretraining treats time-synchronized channels as paired views of one event and learns a shared embedding space with a symmetric contrastive loss. Give each modality \(m\) its own encoder \(f_m\) mapping a window to a unit vector. For a batch of \(N\) synchronized instants, modality-A embedding \(z^A_i\) and modality-B embedding \(z^B_i\) from the same instant \(i\) are a positive pair; all mismatched instants are negatives. The InfoNCE loss, run in both directions and averaged, is

$$\mathcal{L} = -\frac{1}{2N}\sum_{i=1}^{N}\left[\log\frac{\exp(z^A_i\!\cdot z^B_i/\tau)}{\sum_{j}\exp(z^A_i\!\cdot z^B_j/\tau)} + \log\frac{\exp(z^B_i\!\cdot z^A_i/\tau)}{\sum_{j}\exp(z^B_i\!\cdot z^A_j/\tau)}\right].$$

This is exactly the sensor analogue of image-text alignment, with wall-clock synchronization playing the role that human captions play for images. After pretraining, the shared space supports zero-shot cross-modal retrieval (find the PPG segment that matches this accelerometer burst) and, more usefully, graceful missing-modality behavior: because every modality maps into the same space, a downstream head trained on the fused embedding still functions when one sensor drops out, the robustness property studied in depth in Chapter 50. The alignment also opens the door to describing signals in language, the sensor-to-text bridge of Chapter 21.

The listing below is the complete cross-modal contrastive step. Two per-modality encoders produce embeddings for the same synchronized batch; the symmetric InfoNCE above is computed as two cross-entropies over the shared similarity matrix, matching the equation term for term.

import torch
import torch.nn.functional as F

def cross_modal_infonce(z_a, z_b, temperature=0.1):
    # z_a, z_b: (N, D) embeddings; row i of BOTH is the SAME synchronized instant
    z_a = F.normalize(z_a, dim=1)
    z_b = F.normalize(z_b, dim=1)
    logits = z_a @ z_b.t() / temperature          # (N, N): logits[i, j] = A_i . B_j
    targets = torch.arange(z_a.size(0), device=z_a.device)  # positive of i is i
    loss_a2b = F.cross_entropy(logits, targets)   # accel query -> match its PPG
    loss_b2a = F.cross_entropy(logits.t(), targets)  # PPG query -> match its accel
    return 0.5 * (loss_a2b + loss_b2a)            # symmetric, CLIP-style

What's Next

In Section 17.7, we stop assuming a good representation and start measuring one. You will learn the probing toolkit, linear and few-shot probes, retrieval and clustering metrics, and leakage-safe evaluation, that tells you whether all this cross-device, cross-user, cross-modal pretraining actually produced features worth deploying, or merely a loss curve that looked reassuring.