Relative and relational objectives (RelCon)

"Two joggers are not identical and not strangers. Most of the world lives in that gap, and a binary label refuses to see it."

A Graded AI Agent

Why binary positives break on sensor data

The augmentation-based positive works when a single semantic class dominates a window and augmentations are label-preserving. Vision inherited this from crops of one object. Sensor streams violate both assumptions. A ten-second inertial window can contain a gait cycle plus a stumble; jitter or scaling augmentations sometimes cross a semantic boundary (a small time-warp can turn a slow walk into something the downstream head reads as a shuffle). Worse, the negative set is polluted: in any reasonably sized batch of wearable data, several "negatives" are other subjects doing the same activity. InfoNCE will dutifully repel them, injecting label noise that no amount of data cleans up.

The fix is to stop pretending similarity is binary. If we had, for each anchor, a trustworthy ordering of candidates from most to least similar, we could ask the embedding to reproduce that ordering rather than a 0/1 verdict. This turns pretraining into a ranking problem over relationships between distinct real samples, hence "relative" and "relational." The two engineering questions become: (1) where does a reliable similarity ranking come from without labels, and (2) what loss turns a ranking into gradients.

A learnable, label-free similarity measure

RelCon answers question (1) with a learnable distance measure trained before the main encoder. The measure judges how well one motion segment can explain another: given a candidate segment, can it reconstruct the masked portions of the anchor after an alignment that tolerates speed differences? Dynamic time warping supplies the alignment so that two walks at different cadences are recognized as close, and a reconstruction objective supplies the graded score. The result is a function \(d(x, c)\) that is small when \(c\) is a good motif-level match for anchor \(x\) and large otherwise, learned entirely from unlabeled signals. This is the sensor-native analogue of the intuition in Chapter 7 that similar signals share time-frequency structure, but here the notion of "similar" is fit to the data rather than fixed by hand.

For each anchor we sample \(K\) candidates from the corpus and score them with \(d\), yielding an ordering \(d_{(1)} \le d_{(2)} \le \dots \le d_{(K)}\). Nearest candidates are soft positives; farthest are clear negatives; the middle is exactly the graded region binary methods throw away.

The relative contrastive loss

Question (2) is answered by a nested InfoNCE. Order the candidates by learned distance. Then, for every rank \(j\), treat candidate \(j\) as the positive and all candidates at least as far as \(j\) as its negatives. Summing these nested terms gives

where \(z\) is the anchor embedding, \(z_{(k)}\) is the embedding of the \(k\)-th nearest candidate, \(s\) is cosine similarity, and \(\tau\) is the temperature. Each term is an ordinary contrastive loss; stacking them forces \(s(z, z_{(1)}) > s(z, z_{(2)}) > \dots\), so the embedding geometry mirrors the similarity ranking. A single candidate is simultaneously a positive (against those farther) and a negative (against those nearer), which is precisely the relational bookkeeping a binary loss cannot express.

import torch
import torch.nn.functional as F

def relative_contrastive_loss(z_anchor, z_cands, dist, tau=0.1):
    # z_anchor: (D,)  z_cands: (K, D)  dist: (K,) learned distances to anchor
    z_anchor = F.normalize(z_anchor, dim=-1)
    z_cands  = F.normalize(z_cands, dim=-1)
    sim = (z_cands @ z_anchor) / tau            # (K,) cosine sim, scaled
    order = torch.argsort(dist)                 # nearest (smallest dist) first
    sim = sim[order]
    losses = []
    for j in range(sim.shape[0] - 1):
        # candidate j is positive; all farther candidates (k >= j) are negatives
        denom = torch.logsumexp(sim[j:], dim=0)
        losses.append(denom - sim[j])
    return torch.stack(losses).mean()

# toy check: distances that agree with embedding sims give a lower loss
z  = torch.randn(64)
zc = torch.randn(8, 64)
d  = torch.arange(8.0)                          # candidate 0 is "closest"
print(float(relative_contrastive_loss(z, zc, d)))

The loop above is written for clarity; the reference implementation batches anchors and computes all nested denominators with a cumulative logsumexp from the far end, so the whole loss is a handful of tensor ops. The temperature and the candidate count \(K\) are the two knobs that matter: too small a \(K\) and the ranking is noisy, too large and most candidates are trivially far and contribute nothing.

When relational objectives are the right tool

Reach for a relative or relational loss when three conditions hold: labels are absent, the corpus has heavy class overlap (many subjects or devices repeating the same motions, so false negatives are rife), and you can define a cheap, trustworthy similarity between distinct samples. Inertial and physiological streams (Chapter 23) fit perfectly. Nearest-neighbor variants such as NNCLR are a lighter-weight cousin: they promote the batch's nearest neighbor to a positive instead of ranking a whole candidate set, which captures relational structure without a learned distance measure but gives up the graded ordering. When the corpus is genuinely single-class per window and augmentations are safe, the extra machinery buys little and plain SimCLR-style contrastive learning is simpler and just as good. And as always in this book, evaluate on a leakage-safe split: because relational objectives explicitly pull same-activity samples together, a subject leaking across the train/test boundary can inflate scores dramatically, so hold out whole subjects and devices per Chapter 5.

What's Next

In Section 17.6, we scale the relational idea across boundaries: pretraining that spans different devices, different users, and different modalities at once, where the notion of "similar" must survive a change of sensor, sampling rate, and body location.