Temporal predictive coding and TS2Vec

"The future is the only label a sensor stream hands you for free, and it always arrives on time."

A Forward-Looking AI Agent

This section assumes you are comfortable with the contrastive machinery of Section 17.2 (the InfoNCE loss, positive and negative pairs) and with the dilated temporal convolutional encoders of Chapter 14. If sampling rate and alignment are hazy, revisit Chapter 3, because both methods slice the same stream twice and must line up timestamps exactly.

Contrastive predictive coding: predict the latent, not the value

What. Contrastive Predictive Coding (CPC, van den Oord et al., 2018) trains an encoder by predicting future latent representations from past context. An encoder \(g_{enc}\) maps each input window to a latent \(z_t\). An autoregressive summarizer \(g_{ar}\) (a GRU or causal convolution) compresses the past into a context vector \(c_t = g_{ar}(z_{\le t})\). From \(c_t\) the model predicts the latent \(k\) steps ahead, and the objective asks it to score the true future latent higher than a bag of distractors sampled from elsewhere.

Why not just forecast the raw signal? Predicting raw accelerometer or ECG values forces the network to spend capacity on high-frequency noise, quantization, and unpredictable detail that carries no semantics. By contrasting in latent space the model only has to capture what makes the true future distinguishable from random negatives, which is precisely the slow, shared structure (gait phase, cardiac rhythm, machine regime) you want a representation to hold. This is mutual-information maximization between context and future, formalized by the InfoNCE bound:

where \(W_k\) is a learned prediction head for horizon \(k\) and \(\mathcal{N}\) contains the positive future latent plus negatives drawn from other timesteps and other sequences in the batch. Minimizing this loss maximizes a lower bound on \(I(c_t; z_{t+k})\).

TS2Vec: hierarchical contextual consistency

What. TS2Vec (Yue et al., AAAI 2022) is a universal representation framework that produces a vector for every timestamp of a series, at every temporal scale, from one contrastive pretraining run. Its central idea is contextual consistency: the representation of a given timestamp should be stable across two different surrounding contexts of the same series. If timestamp \(t\) means the same thing whether you saw it inside window A or window B, then its embedding has captured the timestamp, not the accident of framing.

How it builds the two contexts. TS2Vec takes one input series and generates two overlapping views by (1) sampling two random crops that share an overlapping region, and (2) applying independent timestamp masking (randomly zeroing input positions before the encoder). A dilated-convolution encoder maps each view to a per-timestamp latent sequence. On the overlap, the same timestamp appears in both views; those paired latents are the positives.

The two-axis loss. TS2Vec contrasts along two axes at once, which is what makes the representation multipurpose:

• Temporal contrast: for a fixed series, the two views of the same timestamp are a positive pair; other timestamps of the same series are negatives. This teaches within-series discriminability, which powers forecasting and anomaly detection.
• Instance-wise contrast: at a fixed timestamp, the two views of the same series are positive; the same timestamp in other series in the batch are negatives. This teaches across-series identity, which powers classification and clustering.

Why hierarchical. After computing the loss at full resolution, TS2Vec max-pools the latent sequence along time by a factor of two and recomputes both losses, repeating until a single vector remains. The same objective therefore shapes the representation at second-scale, minute-scale, and whole-recording scale. To get an instance embedding you max-pool the final timestamp vectors; to get a per-sample embedding for a change detector you read the bottom level directly. One pretraining, every granularity.

Implementing the contextual-consistency loss

The core of TS2Vec is short: encode two masked crops, align them on their overlap, and sum a temporal and an instance-wise InfoNCE term over a hierarchy of pooled scales. The snippet below shows the dual loss at a single scale so the two axes are explicit; the full method wraps it in a max-pool loop.

import torch
import torch.nn.functional as F

def dual_contrastive_loss(z1, z2):
    # z1, z2: (batch B, time T, dim C) latents for the two views,
    # already aligned on the shared overlap region.
    B, T, C = z1.shape
    loss = 0.0

    # --- Instance-wise contrast: fix a timestamp, contrast across series ---
    zi = torch.cat([z1, z2], dim=0)            # (2B, T, C)
    zi = zi.transpose(0, 1)                     # (T, 2B, C)
    sim = torch.matmul(zi, zi.transpose(1, 2))  # (T, 2B, 2B)
    logits = -F.log_softmax(sim, dim=-1)
    # positive of row i is its counterpart view at offset B
    idx = torch.arange(B, device=z1.device)
    inst = (logits[:, idx, idx + B].mean() + logits[:, idx + B, idx].mean()) / 2
    loss += inst

    # --- Temporal contrast: fix a series, contrast across timestamps ---
    zt = torch.cat([z1, z2], dim=1)            # (B, 2T, C)
    sim = torch.matmul(zt, zt.transpose(1, 2))  # (B, 2T, 2T)
    logits = -F.log_softmax(sim, dim=-1)
    t = torch.arange(T, device=z1.device)
    temp = (logits[:, t, t + T].mean() + logits[:, t + T, t].mean()) / 2
    loss += temp
    return loss / 2

As always with self-supervised pretraining, the split that guards against optimistic numbers lives outside this loss. Pretrain and evaluate on disjoint recordings, subjects, and devices, never on shuffled windows from the same trace, or timestamp-level leakage will inflate every downstream score. The leakage-safe partitioning discipline of Chapter 5 applies here in full.

When to choose predictive coding versus TS2Vec

When. Reach for CPC-style predictive coding when your downstream task is intrinsically about the future or about long-range temporal order: forecasting, next-event prediction, or streaming inference where a causal context vector \(c_t\) is exactly the online state you want to carry forward (see Chapter 60). Reach for TS2Vec when you need one embedding that serves many downstream tasks at many granularities, especially when you cannot predict in advance whether you will do classification, retrieval, or point-level anomaly scoring. In practice TS2Vec is the stronger default for mixed sensor workloads because its instance axis and its per-timestamp output cover both ends of the granularity range that a single CPC context vector does not.

What's Next

In Section 17.5, we move from "these two views should agree" to a subtler question: how should representations encode the relative distance between samples rather than a binary same-or-different verdict? We meet relative and relational objectives, and RelCon, which learn a graded notion of similarity that softens the hard-negative assumption these consistency methods rely on.