Benchmarks (HAROOD, BenchHAR, WILDS)

"Every method I met claimed to beat distribution shift. Then I checked whether the same subject was in both the train and the test split, and most of the miracles evaporated."

A Skeptical AI Agent

What a shift benchmark must specify before it reports a number

An ordinary benchmark answers "how accurate is the model on held-out data drawn from the same distribution as training." A shift benchmark must answer something stricter: "how accurate is it on data drawn from a distribution the model has never seen, and does it know when it is guessing." That requires four commitments the benchmark author has to make explicit. First, the shift axis: the metadata field along which train and test are separated. In sensing this is almost always subject, device, body position, or recording site, never a random row shuffle. Second, the split protocol: leave-one-subject-out (LOSO), leave-one-device-out, or a fixed OOD test split with a disjoint validation domain used for model selection. Third, the metric set: not just mean accuracy but the worst-performing group, because a fleet is only as trustworthy as its unluckiest deployment. Fourth, the adaptation budget: whether test-time adaptation is allowed, how many unlabeled target samples it may see, and whether it adapts online or offline. A benchmark that leaves any of these implicit is a benchmark you can beat by accident.

WILDS: making the domain a first-class citizen

WILDS (Koh et al., 2021) is the framework that reframed distribution-shift evaluation for the whole field. Its contribution is less any single dataset than a discipline: every example carries a domain label (which hospital, which camera trap, which region, which time period), and the benchmark ships an official split in which the test domains are disjoint from the training domains. It distinguishes two shift structures that sensing engineers meet constantly: domain generalization, where you must perform on domains never seen in training, and subpopulation shift, where all groups appear in training but their proportions change at test time. Crucially WILDS reports worst-group or worst-domain performance alongside the average, which is the number that matters when a model will be deployed across a fleet of heterogeneous sensors. Although WILDS's headline datasets are vision and text, its protocol is exactly what a sensor benchmark needs, and the modern HAR benchmarks below are best understood as WILDS's discipline applied to inertial and physiological streams.

BenchHAR: cross-subject, cross-device, cross-dataset HAR

BenchHAR names the unified evaluation protocol for sensor-based human-activity recognition: a common set of public datasets (the lineage runs through UCI-HAR, PAMAP2, Opportunity, DSADS, USC-HAD, and the heterogeneity-focused HHAR), a common windowing and preprocessing pipeline, and, above all, a common set of leakage-safe splits. It evaluates along three shift axes that a wearable actually experiences. Cross-subject (leave-one-subject-out) probes covariate shift from gait and body morphology. Cross-device probes the sampling-rate, sensitivity, and mounting differences that HHAR was built to expose, where a model trained on one phone model quietly fails on another. Cross-dataset is the harshest: train on one study's protocol and sensors, test on another's, which folds in label-definition drift as well as covariate shift. The value of a named protocol is comparability: when two adaptation methods both report BenchHAR cross-device F1, the numbers mean the same thing, which is precisely what ad-hoc per-paper splits destroyed. This is the protocol Lab 66 builds on, and it is the natural evaluation home for the DIVERSIFY, TENT, and optimization-free methods of Sections 66.3 through 66.5.

HAROOD: turning HAR into an open-world detection task

Closed-set HAR assumes every test window belongs to one of the training classes. Deployed wearables violate this hourly: the wearer does something the taxonomy never covered (climbing a ladder, swimming, simply fidgeting), and a closed-set model must, by construction, assign it a known label with often high confidence. HAROOD reframes HAR as an out-of-distribution detection problem: the benchmark holds out entire activity classes as "unknown," and the model is scored not only on classifying the known activities but on abstaining on the unknown ones. Its metrics are the OOD-detection standard from Section 66.2: AUROC for separating in-distribution from out-of-distribution windows, and FPR@95TPR, the false-positive rate when the detector is tuned to catch 95% of true novelties. This matters for safety: a fall detector that confidently classifies an unseen motion as "walking" is worse than one that says "I do not know." HAROOD is where the OOD scores of this chapter get their honest report card.

Computing the numbers that actually matter

Three metrics carry most of the weight in these benchmarks, and none of them is mean accuracy. Code 66.7.1 computes all three from a scored evaluation set: worst-group accuracy (the WILDS/BenchHAR discipline), and the AUROC plus FPR@95TPR pair that HAROOD uses to score abstention. Reading it makes the contract concrete: the OOD score is just a per-window confidence, and the benchmark is a ranking problem over those scores.

import numpy as np
from sklearn.metrics import roc_auc_score

def worst_group_accuracy(y_true, y_pred, group):
    """Min over domains (subject/device) of per-group accuracy: the fleet's unluckiest sensor."""
    accs = []
    for g in np.unique(group):
        m = group == g
        accs.append((y_pred[m] == y_true[m]).mean())
    return min(accs), np.mean(accs)          # worst-group, and average for contrast

def fpr_at_95_tpr(id_score, ood_score):
    """OOD detection: higher score = more in-distribution. FPR when 95% of true OOD is caught."""
    y = np.r_[np.ones_like(id_score), np.zeros_like(ood_score)]   # 1 = in-dist
    s = np.r_[id_score, ood_score]
    auroc = roc_auc_score(y, s)
    thr = np.quantile(ood_score, 0.95)       # threshold catching 95% of OOD (TPR_ood)
    fpr = (id_score < thr).mean()            # in-dist windows wrongly flagged as OOD
    return auroc, fpr

wg, avg = worst_group_accuracy(y_true, y_pred, subject_id)
auroc, fpr95 = fpr_at_95_tpr(id_conf, ood_conf)
print(f"worst-group {wg:.3f}  avg {avg:.3f}  |  OOD AUROC {auroc:.3f}  FPR@95 {fpr95:.3f}")

The gap that Code 66.7.1 exposes between worst-group and average accuracy is the number a fleet owner cares about, because customers experience the worst group, not the mean. A method that lifts the average by two points while dropping the worst group by five has made your product less trustworthy, and only a benchmark that reports both will tell you.

How these benchmarks lie, and how to read them honestly

A benchmark is only as trustworthy as the leakage it prevents and the metric it foregrounds, so a few failure modes are worth naming. Validation-domain leakage: selecting hyperparameters on the test domains, which quietly turns domain generalization into transduction; WILDS forbids it by giving you a disjoint OOD-validation split, and you should use it. Overlapping windows across the split boundary: sliding windows from one recording bleeding across a subject cut, re-introducing the leakage the split was meant to remove. Cherry-picked shift axis: reporting cross-subject where the deployment shift is actually cross-device, so the easy axis flatters the method. Average-only reporting: hiding a collapsed worst group inside a healthy mean. And for test-time adaptation specifically, online evaluation order matters: adapting on class-sorted or tiny non-i.i.d. batches, the pitfall the TTAB benchmark was built to expose, can make an adaptation method look either magical or catastrophic depending on stream order alone. The defense is the same discipline throughout this part of the book: fix the shift axis to match the deployment, hold out the domain, report the worst group next to the average, and treat any result whose split you cannot reconstruct as unproven.

What's Next

In Chapter 67, we stop asking whether a model is wrong and start asking why. Once a benchmark tells you the worst group collapsed under device shift, interpretability and root-cause analysis for time series let you trace that collapse back to a specific channel, frequency band, or window, so the next fix is aimed rather than guessed.