User/site/device/session splits

"You did not test whether I recognize walking. You tested whether I remember Subject 7's ankles."

A Recently Benchmarked AI Agent

This section assumes you already treat leakage as a first-class hazard from Chapter 5 and that you distinguish task metrics from operational metrics as in Section 65.1. Here we make one hazard concrete and mechanical: identity leakage, the contamination that occurs when samples sharing a grouping key are split across train and test. We write \(g(i)\) for the group key of sample \(i\) (its user, site, device, or session id) and require, for a clean split, that no group appears on both sides: \(\{g(i) : i \in \text{train}\} \cap \{g(i) : i \in \text{test}\} = \varnothing\).

Why a random split leaks in sensor data

Under the standard learning-theory story, a held-out test set estimates generalization because the samples are independent and identically distributed. Sensor samples violate independence by construction. Two accelerometer windows 20 milliseconds apart are nearly the same window; two ECG beats from one patient share that patient's conduction geometry; two vibration spectra from one pump share its exact bearing clearances. The label you care about (activity, arrhythmia, fault) is entangled with a much stronger nuisance signal: who or what produced the measurement. A high-capacity model will happily exploit the nuisance, because during a random split the nuisance is predictive: the same subject's other windows sit in the test set wearing the same fingerprint.

The result is a benchmark that rewards the wrong skill. A human-activity model can hit 98 percent under a random split and fall to 80 percent when it must generalize to a person it has never seen, because the random split let it memorize per-subject idiosyncrasies (limb length, sensor placement, walking style) instead of the activity. The gap between those two numbers is not noise; it is the portion of your reported accuracy that was identity leakage. The fix is not a better model. It is a grouping key applied before the cut.

The identity hierarchy: session, device, user, site

Sensor identities nest. From narrowest to broadest, the four keys you will grip most often are: session (one continuous recording, one mounting, one sitting), device (one physical unit with its own calibration and firmware), user (one person or one asset across many sessions and possibly many devices), and site (one factory, hospital, building, or vehicle fleet, sharing an environment). Each broader key is a stronger test because it forces the model to generalize across more shared nuisance factors at once.

Session split. Group by recording. This kills the most flagrant leak, adjacent-window correlation, but a user recorded across ten sessions still appears in both train and test, so it does not test cross-person generalization. Use it as a floor, never as your headline.

Device split. Group by hardware unit. This isolates per-unit calibration, mounting resonance, and firmware, the factors that made the wind-turbine model in Chapter 2's measurement model drift when a sensor was swapped. Essential when you manufacture at scale and each unit off the line differs.

User split, or leave-one-subject-out (LOSO). Group by person. The gold standard for wearables and biosignals, because physiology and behavior are the dominant nuisance. In Chapter 26 on activity recognition, LOSO is the difference between a publishable claim and a self-deception.

Site split. Group by deployment location. The hardest and most operationally honest test for fleets, because a site bundles environment, population, installation practice, and often a device generation. This is the split regulators and clinical reviewers care about, a thread we pick up in Chapter 34.

The keys are also nested constraints. If you split by user but two users share a device, a device fingerprint still leaks; if you split by site but the same wearer visits two sites, a person leaks. The safe rule is to group by the coarsest identity that co-varies with the nuisance you fear, and when identities cross (one device used by many users), group by the union so no shared key survives the cut.

Grouped cross-validation in practice

A single grouped split gives one estimate with wide error bars, because a handful of held-out users may be lucky or unlucky. Grouped k-fold cross-validation rotates the held-out groups so every group is tested exactly once, then reports the mean and spread across folds. Two library primitives cover almost everything. GroupKFold guarantees that no group crosses a fold boundary. StratifiedGroupKFold does the same while also keeping the class balance roughly equal across folds, which matters when your positive rate is low, the concern of Section 65.4. The code below runs a leave-users-out evaluation and, critically, prints the per-fold spread rather than only the mean, because the variance across users is itself a headline result.

import numpy as np
from sklearn.model_selection import StratifiedGroupKFold, cross_val_score
from sklearn.ensemble import RandomForestClassifier

# X: (n_windows, n_features)  y: activity label  groups: subject id per window
X, y, groups = load_har_windows()          # your feature extractor

cv = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=0)
clf = RandomForestClassifier(n_estimators=300, n_jobs=-1)

# each fold holds out a DISJOINT set of subjects; class balance is preserved
scores = cross_val_score(clf, X, y, groups=groups, cv=cv, scoring="f1_macro")

# sanity check: no subject may appear in both train and test of any fold
for tr, te in cv.split(X, y, groups):
    assert set(groups[tr]).isdisjoint(set(groups[te]))

print(f"subject-out macro-F1: {scores.mean():.3f} +/- {scores.std():.3f}")
print("per-fold:", np.round(scores, 3))     # the spread IS a result

The assertion in that snippet is not decoration. It is the one line that catches the most expensive benchmarking bug in the field, and you should keep an equivalent guard in every evaluation harness you write. Reporting the per-fold spread alongside the mean is equally load-bearing: a mean macro-F1 of 0.88 with a fold standard deviation of 0.02 is a robust model, whereas the same 0.88 with a spread of 0.15 means the system works well for most people and fails badly for some, which is an operational and fairness problem you must surface, not average away.

Choosing the key and reading the gap

Pick the grouping key by naming the entity your model will meet in production that it did not meet in training. Ship to new customers? Split by site. Ship the same watch to millions of new wrists? Split by user. Swap sensors during maintenance? Split by device. Report several splits together, because the gap between them is diagnostic. A small drop from session-out to user-out means the model generalizes across people; a large drop localizes the failure to per-person nuisance and tells you to collect more subjects, not more data per subject. A further drop from user-out to site-out isolates environment and installation effects, which is precisely the distribution shift that Chapter 66 equips you to adapt to. The ladder of splits is a decomposition of your generalization error by cause.

What's Next

In Section 65.3, we add the second axis of leakage that grouping alone cannot fix: time. Even a perfectly user-disjoint split can cheat if a future window trains a model that is tested on the past, so we turn to time-aware cross-validation, forward-chaining, and purged, embargoed folds that respect the one-way arrow of the sensor clock.