Probing and evaluating learned representations

"You spent a week teaching me to embed the world with no labels. Now you want to know if I learned anything. Fair. Hand me twenty labels and stand back."

An Empirically Minded AI Agent

Linear probing: the standard readout

The canonical SSL evaluation freezes the pretrained encoder \(f_\theta\), discards the projection head used during pretraining, and trains a single linear layer (or a plain logistic regression) on top of the frozen embeddings using whatever labels you have. Accuracy of that linear probe is your headline number. The logic is deliberate: a linear readout cannot invent new features, so it measures whether the information a downstream task needs is already present and, crucially, linearly accessible in the frozen representation. If a two-line logistic regression separates activities from your embeddings, the encoder did the representational work; if it cannot, no amount of pretraining spin changes the verdict.

Why freeze rather than fine-tune for evaluation? Because fine-tuning entangles two things you want to measure separately: the quality of the pretrained features and the capacity of the network to adapt them. A mediocre representation can still reach high fine-tuned accuracy if the backbone is large and the labeled set is generous, which flatters a weak objective. The linear probe isolates the representation. In practice you report both: the linear-probe number for a clean comparison of objectives, and the fully fine-tuned number for the accuracy you would actually deploy. A large gap between them is diagnostic. It means the useful information is present but nonlinearly tangled, which is common for masked-reconstruction encoders and is exactly why Section 17.3 models often fine-tune better than they probe.

The label-efficiency curve: the number that matters

A single accuracy figure at one label budget is a snapshot; the story of self-supervision is a curve. Fix the frozen encoder, then fit the probe on \(m\) labeled examples per class for \(m \in \{1, 2, 5, 10, 20, 50, \dots\}\), and plot downstream accuracy against \(m\) on a log axis. Overlay the same curve for a randomly initialized encoder and for a fully supervised model trained from scratch on the same budgets. The vertical gap at small \(m\) is the payoff of pretraining, and the horizontal gap is even more interpretable: read off how many labels the pretrained encoder needs to match the from-scratch model's accuracy at, say, 100 labels per class. A good sensor SSL result looks like "the pretrained encoder reaches the supervised baseline's 200-label accuracy with 20 labels," a tenfold reduction in annotation cost.

This curve is the honest deliverable of Section 17.1's promise that labels are the bottleneck. It also exposes overclaiming: many objectives look identical at 500 labels per class, where everything saturates, and only separate in the 1-to-20 regime where the data is actually scarce. Always sweep down into the low-label regime; that is where SSL either delivers or does not.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

def label_efficiency_curve(Z_tr, y_tr, Z_te, y_te,
                           shots=(1, 2, 5, 10, 20, 50), seed=0):
    """Frozen embeddings in, accuracy-vs-labels curve out.
    Z_* are L2-normalized encoder outputs; splits are SUBJECT-WISE."""
    rng = np.random.default_rng(seed)
    classes = np.unique(y_tr)
    curve = []
    for m in shots:                      # m labeled windows per class
        idx = np.concatenate([
            rng.choice(np.where(y_tr == c)[0], size=m, replace=False)
            for c in classes])
        clf = LogisticRegression(max_iter=2000, C=1.0)
        clf.fit(Z_tr[idx], y_tr[idx])    # linear probe on m*|C| labels
        acc = accuracy_score(y_te, clf.predict(Z_te))
        curve.append((m, acc))
    return curve

The function above is the exact protocol behind Lab 17. Notice that the encoder runs once to produce embeddings; the expensive part of evaluation is therefore trivial, and you can afford to average every point over several random label draws to get error bars, which you should, because the 1-shot point in particular has enormous variance.

Label-free geometry: alignment, uniformity, and collapse

Sometimes you want to judge a representation before you have any labels at all, or diagnose why a probe underperforms. The embedding geometry itself carries signal. For contrastive encoders, Wang and Isola's decomposition gives two complementary intrinsic metrics on \(L2\)-normalized embeddings. Alignment measures how close positive pairs land:

$$\mathcal{L}_{\text{align}} = \mathbb{E}_{(x,x^{+})}\,\lVert f(x) - f(x^{+})\rVert^{2},$$

and uniformity measures how evenly embeddings spread over the hypersphere, penalizing clumping:

$$\mathcal{L}_{\text{unif}} = \log\,\mathbb{E}_{x,y}\,e^{-2\lVert f(x)-f(y)\rVert^{2}}.$$

A healthy encoder scores low on both: positives coincide (good alignment) yet the space is used fully (good uniformity). The pathology these metrics catch is dimensional collapse, where the encoder maps everything into a low-dimensional subspace or, in the worst case, a single point, because a degenerate constant embedding trivially minimizes a naive contrastive loss. You can quantify collapse directly by taking the singular values of the embedding covariance matrix and computing the effective rank or simply plotting the spectrum: a rapidly decaying spectrum with a handful of dominant directions signals collapse, even when the pretraining loss looks fine. This is the label-free early-warning system that a loss curve cannot provide, and it is why Section 17.5's relational objectives and negative-free methods invest so heavily in anti-collapse mechanisms.

Probing fairly: the split is the experiment

Every metric above is meaningless without a leakage-safe split, and sensor data makes leakage unusually easy. If windows from one person land in both the probe-training and test sets, the probe can recognize the person rather than the activity, and your accuracy inflates by tens of points for reasons that will not survive contact with a new user. The rule is subject-wise (or device-wise) grouping: hold out whole subjects, never individual windows, using GroupKFold with the subject id as the group. The same discipline applies to the time axis; adjacent overlapping windows are near-duplicates, so a random split leaks a window's near-twin into test. This is the running leakage-safe thread of the book (Chapter 5), and it bites hardest exactly here, because a strong pretrained encoder is also a strong identity encoder and will happily exploit the shortcut you left open.

Beyond linear probing, two more readouts round out a fair evaluation. \(k\)-NN probing classifies a test embedding by the majority label of its nearest neighbors under cosine distance; it has zero trainable parameters, so it measures the raw geometry with no fitting at all and cannot overfit a small label set. Clustering purity, obtained by \(k\)-means on the embeddings and comparing clusters to true labels via normalized mutual information, tells you whether the classes are separated in the space even before any supervision. And to compare two encoders' internal representations (say, a contrastive and a masked model, or two layers of one model), centered kernel alignment (CKA) gives a single similarity score between representation matrices; it is the standard tool for asking "did these two objectives learn the same thing" and it appears again in the interpretability toolkit of Chapter 67.

What's Next

In Chapter 18, we take the frozen encoder you have now learned to trust and ask a harder question than "is it accurate": is it honest about what it does not know. Probing tells you the representation is usable; calibration and conformal prediction tell you when to believe a specific prediction it makes, how to attach guaranteed coverage to that belief, and when the safest output is to abstain. A label-efficient encoder that cannot flag its own uncertainty is not yet deployable, and that is the gap the next chapter closes.