Contrastive multimodal pretraining

"Teach two senses to point at the same moment in the world, and they will find each other in the dark."

An Aligned AI Agent

Co-occurrence is a label you already have

The what is a change of supervision. Chapter 17 built positives by augmenting one signal (two croppings of the same IMU window are a positive pair). Multimodal contrastive learning takes the two positives from different sensors that recorded the same event. If accelerometer window \(a_i\) and audio window \(b_i\) were captured at the same time on the same device, they are a positive pair; \(a_i\) paired with any other clip's audio \(b_j\) (\(j \neq i\)) is a negative. The why is that this "same event" relation is exactly the alignment fusion needs and is almost never explicitly labeled, yet it is present for free in any multi-sensor log. You are converting time synchronization, a property you get from the recording hardware, into semantic supervision.

The how is CLIP's symmetric InfoNCE objective (Radford et al., 2021), lifted from image-text to any sensor pair. Two encoders \(f\) and \(g\) map a batch of \(N\) paired windows to L2-normalized embeddings \(u_i = f(a_i)\) and \(v_i = g(b_i)\) in the same \(\mathbb{R}^d\). Their cosine-similarity matrix \(S_{ij} = u_i^\top v_j / \tau\) should be large on the diagonal and small off it. A temperature \(\tau\) sharpens the softmax. The loss reads each row as a classification over the batch (which audio matches this IMU?) and each column symmetrically:

$$\mathcal{L} = \tfrac{1}{2N}\sum_{i=1}^{N}\Big[ -\log\frac{e^{S_{ii}}}{\sum_j e^{S_{ij}}} \;-\; \log\frac{e^{S_{ii}}}{\sum_j e^{S_{ji}}} \Big].$$

The when is the label-scarce, pair-rich regime: you have millions of synchronized windows and a few thousand annotations. Pretrain contrastively on the pairs, then attach a small head and fine-tune on the labels. When labels are abundant and pairs are not, skip this and train fusion end-to-end as in Section 50.2.

The three knobs that decide whether it works

Three design choices dominate the outcome, and all three are easy to get wrong. The first is the temperature \(\tau\). Too high and every pair looks equally similar, so no gradient separates positives from hard negatives; too low and the loss fixates on the single hardest negative and destabilizes. CLIP learns \(\tau\) as a parameter but clips it; a fixed \(\tau \approx 0.07\) is a safe start. The second is batch size, because the negatives are the other items in the batch: a batch of 64 gives each anchor 63 negatives, a batch of 32k gives 32{,}767, and contrastive quality rises sharply with more negatives. This is the real reason CLIP-style training is compute-hungry, and it is the knob most often under-provisioned on sensor data.

The third and most sensor-specific is the definition of a positive pair, which is a leakage minefield. A "pair" is two modalities from the same time window, but if your negatives are drawn from the same recording session a few seconds apart, they may be near-identical (a person standing still emits nearly the same IMU and audio second after second), so the model is punished for correctly calling two similar things similar. The fix is to sample negatives across sessions and subjects, and to split train and test by subject exactly as Chapter 5 demands. Otherwise the impressive retrieval accuracy is memorized session identity, not learned semantics.

import torch, torch.nn.functional as F

def clip_loss(u, v, temperature=0.07):
    # u, v: (N, D) embeddings of paired windows from two modalities
    u = F.normalize(u, dim=-1)                 # unit sphere
    v = F.normalize(v, dim=-1)
    logits = (u @ v.T) / temperature           # (N, N) similarity matrix
    targets = torch.arange(u.size(0), device=u.device)  # diagonal is the match
    loss_a = F.cross_entropy(logits, targets)          # each IMU picks its audio
    loss_b = F.cross_entropy(logits.T, targets)        # each audio picks its IMU
    return 0.5 * (loss_a + loss_b)

# retrieval sanity check: is the true match the nearest neighbour?
with torch.no_grad():
    u = F.normalize(torch.randn(256, 128), dim=-1)     # stand-in embeddings
    v = F.normalize(torch.randn(256, 128), dim=-1)
    r1 = ((u @ v.T).argmax(dim=1) == torch.arange(256)).float().mean()
    print(f"top-1 cross-modal retrieval: {r1.item():.3f}")  # ~0.004 at random

As Listing 50.5 shows, the loss itself is trivial; the engineering is entirely in the encoders that feed it and the batch that populates its negatives. Top-1 cross-modal retrieval is the pretraining metric to trust, because it measures alignment directly without needing a single downstream label.

What contrastive pretraining does not give you

The method has real limits worth naming before you reach for it. Contrastive alignment learns what is shared between modalities and is happy to discard what is not. If the microphone hears an alarm the accelerometer cannot feel, that alarm is modality-specific information the contrastive objective actively pushes out of the shared space, because it cannot be predicted from the other stream. When your downstream task depends on such private information, a purely contrastive representation will underperform a model that also keeps a reconstruction or masked-prediction term (the direction of joint contrastive-plus-generative objectives). Second, the shared space is only as aligned as your pairs are truly synchronous: a fixed 200 ms offset between the two sensors, unremoved, teaches the encoders to align the wrong instants, which is why the timing hygiene of Chapter 3 is not optional here. Third, contrastive pretraining aligns; it does not calibrate. The cosine similarities are not probabilities, and downstream confidence still needs the treatment of Chapter 18.

from lightly.loss import NTXentLoss     # symmetric InfoNCE, multi-GPU negatives
criterion = NTXentLoss(temperature=0.07, gather_distributed=True)
loss = criterion(imu_emb, audio_emb)   # any two Section 50.1 encoders

What's Next

In Section 50.6, we turn from building fused representations to seeing inside them, using attention rollout, modality-ablation, and attribution methods to answer which sensor actually drove a given decision, so that a fusion model's confidence can be traced back to the evidence that earned it.