Part IV: Deep Learning for Sensor Time Series
Chapter 17: Self-Supervised and Contrastive Sensor Learning

Contrastive learning with sensor augmentations

"Show me the same walk twice, once with the watch loose and once tightened, and I will learn that both are walking. Nobody had to tell me the word."

A Self-Supervised AI Agent

Prerequisites

This section assumes you accept the premise of Section 17.1: labels are scarce but raw sensor windows are nearly free. It builds directly on the augmentation catalogue for sensor streams from Chapter 13 (jitter, scaling, time warping, channel dropout) and on any encoder that turns a window into a vector, whether the 1D CNN of Chapter 14 or the transformer of Chapter 15. You need the softmax and cross-entropy from Appendix B, cosine similarity, and the leakage-safe splitting discipline of Chapter 5.

The Big Picture

Contrastive learning is a way to teach an encoder useful structure without a single label, using one deceptively simple instruction: an example and a distorted copy of itself should land close together in embedding space, while two unrelated examples should land far apart. The trick is that you get to define "a distorted copy" through augmentations, and for sensors those augmentations encode real physics: the same activity recorded with the watch worn tighter, the same vibration with a little more amplifier noise, the same gait sampled a beat faster. By declaring which changes are irrelevant, you implicitly declare what the representation must be invariant to, and the network discovers the rest. Done well, a contrastive encoder pretrained on millions of unlabeled windows needs only a few dozen labels per class to match a fully supervised model, because the hard part, learning what "different activity" even means, is already finished. Done carelessly, one bad augmentation quietly deletes the signal you were trying to predict.

The objective: pull positives together, push negatives apart

Fix an encoder \(f_\theta\) that maps a window \(x\) to an embedding, followed by a small projection head \(g\) that maps it to a unit vector \(z=g(f_\theta(x))/\lVert\cdot\rVert\). Take a batch of \(N\) windows. Augment each one twice with independently sampled transforms, giving \(2N\) views. The two views of the same window are a positive pair; every other view in the batch is a negative. For a positive pair \((i,j)\) the normalized temperature-scaled cross-entropy loss, NT-Xent, is

$$\ell_{i,j} = -\log \frac{\exp(\operatorname{sim}(z_i,z_j)/\tau)}{\sum_{k=1}^{2N}\mathbb{1}_{[k\neq i]}\exp(\operatorname{sim}(z_i,z_k)/\tau)},$$

where \(\operatorname{sim}(u,v)=u^{\top}v\) is cosine similarity on the unit sphere and \(\tau\) is a temperature, typically 0.1 to 0.5. Read it as a softmax classification: given anchor \(i\), pick its true partner \(j\) out of the \(2N-1\) candidates. Minimizing the loss maximizes agreement between views of the same underlying signal relative to everything else in the batch. This is the SimCLR recipe, and it transfers to time series almost unchanged; what changes, and what this section is about, is the augmentation family.

Key Insight

Contrastive learning does not learn "what a signal is." It learns "what a signal is up to the augmentations you chose." The augmentation set is the entire inductive bias. If you include time warping as a positive-generating transform, you are asserting that speed is nuisance, which is right for activity recognition but catastrophic for cadence or heart-rate estimation, where speed is the label. Choosing augmentations is therefore not a preprocessing detail; it is where you inject domain knowledge, and it deserves the same scrutiny as choosing a loss.

Augmentations that respect sensor physics

A useful augmentation must be semantics-preserving: it should change things a downstream label does not care about, and leave the label intact. Because sensor signals come with a known measurement model (Chapter 2), you can reason about this precisely rather than guessing. The workhorse family for inertial and physiological streams:

Two augmentations composed usually beat one, because a single transform can be inverted or ignored, while a random composition presents a genuinely harder invariance to learn. Empirically, the jitter-plus-scaling and permutation-plus-jitter pairs are strong defaults for human-activity data; rotation is the highest-value addition when your sensor is body-worn and mounting is uncontrolled.

The augmentation that erases your label

Every augmentation is a claim about invariance, and a wrong claim leaks label information away silently, with no error message. Time warping applied to a photoplethysmography stream destroys the exact quantity a heart-rate model needs. Aggressive scaling erases the amplitude that separates a gentle tap from a hard press on a tactile sensor. Permutation shreds the P-QRS-T ordering that an ECG classifier depends on (Chapter 29). The failure is invisible during pretraining, where the loss still drops nicely, and only surfaces as a disappointing fine-tuning curve. Always ask: does my downstream label survive this transform? If not, remove it.

Negatives, batch size, and the false-negative trap

The denominator of NT-Xent treats every other window in the batch as a negative, so the number of negatives is roughly the batch size. More negatives sharpen the contrast and generally help, which is why contrastive methods historically wanted large batches or a memory queue of past embeddings (the MoCo design). But sensor batches hide a subtlety: because activities are repetitive and classes are few, two different windows in the same batch are often the same activity. Treating them as negatives, a false negative, actively pushes apart embeddings that should be neighbors. This matters far more for a five-class activity dataset than for a thousand-class image set. Mitigations include drawing batches so that windows from the same subject-and-session are unlikely to collide, keeping the temperature \(\tau\) from becoming too small (which over-penalizes hard negatives), and using approaches that lean less on negatives at all, the predictive and relational objectives of Sections 17.4 and 17.5. And as always, positives must never straddle a train/test boundary: augment after splitting, or you leak (Chapter 5).

The listing below implements the NT-Xent loss for a batch of \(2N\) stacked embeddings and a single sensor augmentation. It is the computational heart of every method in this section; the prose above describes exactly what each line enforces.

import torch
import torch.nn.functional as F

def nt_xent(z, temperature=0.2):
    # z: (2N, D), rows 0..N-1 are view-A, rows N..2N-1 are view-B of the SAME windows
    z = F.normalize(z, dim=1)
    N = z.shape[0] // 2
    sim = z @ z.t() / temperature                 # (2N, 2N) cosine sims
    sim.fill_diagonal_(float("-inf"))             # an anchor is never its own positive
    # positive of row i is its counterpart N rows away (and vice versa)
    targets = torch.arange(2 * N, device=z.device)
    targets = (targets + N) % (2 * N)
    return F.cross_entropy(sim, targets)          # softmax "find your partner"

def jitter_scale(x, sigma=0.05, scale=0.1):
    # x: (B, C, T) accelerometer window; two independent draws give a positive pair
    x = x + sigma * torch.randn_like(x)                       # thermal-noise mimic
    x = x * (1 + scale * torch.randn(x.size(0), x.size(1), 1, device=x.device))
    return x                                                   # gain-drift mimic
NT-Xent contrastive loss and a jitter-plus-scaling augmentation. Two independent calls to jitter_scale on the same batch produce the positive pair; stacking both views and passing the encoder output to nt_xent completes one SimCLR step. The fill_diagonal_ and modular targets lines are where "same window = positive, everything else = negative" is enforced.

Practical Example: label-scarce gait monitoring on a fitness band

A wearables team has 40,000 hours of tri-axial accelerometer data from a fitness band but only 300 short, expert-annotated clips of six activities (walk, run, stairs, cycle, sit, stand). Training a supervised CNN on 300 clips overfits badly and collapses across users. Instead they pretrain a 1D-CNN encoder contrastively on two-second windows sampled from the full 40,000 hours, using rotation plus jitter plus scaling as the augmentation set, chosen because the band's orientation on the wrist is uncontrolled and gain varies across manufacturing lots. After pretraining with no labels, they freeze the encoder and fit a linear classifier on the 300 clips. It reaches within two points of the accuracy a supervised model needs 15x more labels to hit, and, because rotation invariance was baked in, it generalizes to users who wear the band upside down, a failure mode the supervised baseline never survived. This label-efficiency curve is exactly what Lab 17 asks you to reproduce, and it connects to the activity-recognition pipeline of Chapter 26.

Right Tool: don't hand-roll the contrastive scaffold

Written from scratch, a correct contrastive trainer is roughly 250 to 400 lines once you add the projection head, the two-view collate function, gradient accumulation for large effective batches, a memory queue, and mixed precision. Libraries such as lightly (SimCLR, MoCo, BYOL heads and losses) or tsai (time-series-first, with the TS-TCC and TS2Vec recipes prewired) reduce this to about 20 lines: you supply the encoder and the augmentation callable, and the library owns the view generation, the NT-Xent bookkeeping, the queue, and the optimizer schedule. Keep the augmentation choice in your own hands, that is the domain knowledge, and let the library own the plumbing.

Research Frontier: where sensor contrastive learning stands

SimCLR and MoCo established the image recipe; the sensor-native line adapts it to temporal structure. TS-TCC (Time-Series Temporal and Contextual Contrasting) pairs a strong-versus-weak augmentation view with a cross-view temporal prediction task and remains a strong human-activity baseline. TS2Vec (covered next, in Section 17.4) contrasts at multiple time scales and hierarchical resolutions rather than over a single global embedding. The frontier question is no longer "does contrastive pretraining help" (it clearly does) but "which augmentations are physically valid per modality," with recent work learning or searching the augmentation policy instead of fixing it by hand, and blending contrastive with masked objectives so a single encoder captures both invariance and reconstruction. For deployment, the same encoder increasingly feeds the cross-device and cross-user pretraining of Section 17.6.

Exercise

Take a public accelerometer HAR dataset. (1) Pretrain a small 1D-CNN encoder with NT-Xent using jitter-plus-scaling, then fit a linear probe on 1%, 5%, and 25% of the labels; plot the label-efficiency curve against a supervised model trained on the same fractions. (2) Now add time warping to the augmentation set and repeat. (3) Replace the activity label with a cadence regression target and rerun step 2. Explain, using the invariance argument from this section, why time warping helped in step 2 but hurt in step 3.

Self-Check

  1. In NT-Xent, what exactly plays the role of the "class" that the softmax is trying to predict, and how many candidates does it choose among for a batch of \(N\) windows?
  2. Why does a false negative hurt a 5-class activity contrastive run more than a 1000-class image run, and name one way to reduce it?
  3. You want a representation for respiratory-rate estimation from a chest-strap signal. Which two augmentations from this section would you refuse to use, and why?

What's Next

In Section 17.3, we drop the need for negatives and augmentation pairs altogether and instead mask out chunks of the window, training the encoder to reconstruct what it cannot see. Masked autoencoding for time series trades the "pull-and-push" geometry of contrastive learning for a generative one, sidesteps the false-negative trap entirely, and turns out to pair beautifully with the channel-masking augmentation you just met here.