Device, user, site, session, and machine splits

"The test set exists to answer one question: what happens when something new walks in the door. If the door already knows the visitor, you asked a different question and reported the wrong answer."

A Disciplined AI Agent

This section assumes you have already cut streams into windows (Section 5.2) and understand the leakage mechanisms catalogued in Section 5.3. Here we turn the diagnosis into a construction procedure. We take a windowed, labeled dataset carrying entity identifiers and produce train, validation, and test partitions whose accuracy estimate matches the deployment question you actually care about.

The grouping unit is a modeling decision, not a default

What. A grouping unit is the entity across whose boundary your model must generalize at deployment. Every window belongs to exactly one value of that unit (this wearer, that machine, this recording session), and a group-disjoint split guarantees that all windows sharing a group value land entirely in train or entirely in test, never split between them. Why. Sensor data is riddled with nuisance signals that are constant within an entity and irrelevant to the label: a specific accelerometer's bias, one person's gait idiosyncrasy, a particular lathe's bearing hum, a room's acoustic reverberation. A random split scatters an entity's windows across train and test, so the model can lower its loss by recognizing the entity rather than the phenomenon. That shortcut evaporates the instant a genuinely new entity appears.

How. Pick the unit by asking a single question: at deployment, what will be new? The answer names your holdout axis. The common units, roughly ordered from finest to coarsest, are:

• Session (recording). One continuous capture from one entity. Holding out sessions defeats the crudest leakage (adjacent overlapping windows) but still lets the same person or device appear in train and test.
• Device. One physical sensor unit. Device-disjoint splits test whether the model survives manufacturing tolerance, mounting variation, and per-unit calibration drift (the subject of Section 5.5).
• User / subject. One person wearing or operating the sensor. Subject-disjoint splitting is mandatory for any wearable, biosignal, or activity model, because human physiology and behavior are the dominant nuisance variables.
• Machine. One physical asset in industrial monitoring: a pump, a turbine, a CNC spindle. Fault models must generalize to machines never seen during training.
• Site. A whole location: a hospital, a factory floor, a building. Site-disjoint splits probe environmental and installation shift, the coarsest and usually hardest generalization gap.

Nested entities and the strictest-boundary rule

Real datasets carry several nested identifiers at once. A wearable study has subject ⊃ device (a loaner watch) ⊃ session (each day's wear) ⊃ window. These nest: every session belongs to one device, every device to one subject. The rule: split at the coarsest boundary your deployment claim requires, and the finer boundaries take care of themselves automatically, because holding out a subject also holds out all of that subject's devices and sessions.

The danger is a partial identifier that silently couples the two sides. If one subject wore two devices and you split by device, that subject's physiology now appears in both partitions even though no device does; you have a subject leak hiding inside a clean-looking device split. Formally, a split into train set \(\mathcal{T}\) and evaluation set \(\mathcal{E}\) is group-safe for unit \(g\) when their group sets are disjoint:

When several units matter at once (say subject and site), the safe construction holds out on the composite key, or splits on the coarsest unit that dominates the others. Verify the intersection is empty for every unit you claim to generalize across, not just the one you sorted on.

Constructing grouped splits in practice

The mechanics are short once the group vector exists. You never shuffle windows; you shuffle groups, then gather every window of the chosen groups. For cross-validation you want each fold's test groups disjoint from its training groups, which is exactly what grouped K-fold provides. The snippet below shows a manual subject-disjoint split and the grouped-CV equivalent, and prints the group overlap so you can prove disjointness rather than assume it.

import numpy as np
from sklearn.model_selection import GroupKFold, GroupShuffleSplit

# X: (N, L, C) windows; y: (N,) labels; groups: (N,) subject id per window
rng = np.random.default_rng(0)
N = 2000
groups = rng.integers(0, 20, size=N)      # 20 subjects
y = rng.integers(0, 3, size=N)

# One held-out split: whole subjects go to test, none shared
gss = GroupShuffleSplit(n_splits=1, test_size=0.25, random_state=0)
train_idx, test_idx = next(gss.split(np.zeros(N), y, groups))
overlap = set(groups[train_idx]) & set(groups[test_idx])
print("shared subjects:", len(overlap))    # must be 0

# K-fold where every fold is subject-disjoint
for k, (tr, te) in enumerate(GroupKFold(n_splits=5).split(np.zeros(N), y, groups)):
    assert not (set(groups[tr]) & set(groups[te]))
    print(f"fold {k}: {len(set(groups[te]))} test subjects, {len(te)} windows")

Two refinements matter for honest numbers. First, groups are rarely balanced (one subject may contribute ten times the windows of another), so a group-disjoint split can accidentally load all of one class onto one side; use stratified-grouped variants or report per-group performance to catch it. Second, with few groups the estimate is high-variance: leaving out one of five machines makes the score swing wildly, so prefer leave-one-group-out cross-validation and report the spread, not a single fold. This ties directly to the small-sample uncertainty tools from Chapter 4.

When a pure entity split is not enough

When to go further. Group-disjoint splitting removes identity leakage, but two related traps survive it. If your data has a strong temporal structure (a machine degrades over months), you may also need the test period to come after the training period so you do not train on the future, combining a machine split with a temporal cutoff. And if the label is correlated with the group (all positive cases came from one site), even a clean split can hand the model a spurious site-equals-label shortcut. The defense is to check that each class is represented across multiple groups on both sides, and to prefer coarser or composite grouping when a nuisance variable and the label are confounded. These distribution-shift concerns are the running theme of Chapter 66, and the leakage-safe benchmarking discipline is formalized in Chapter 65.

Group-aware splitting is deceptively simple to implement and decisive in effect: the same model and the same data can post a triumphant or a sober number depending only on where you draw the boundary. Draw it at the entity your deployment must generalize across, prove the groups are disjoint, and your accuracy finally means what it says.

What's Next

In Section 5.5, we tackle the nuisance that made device and subject splits necessary in the first place: per-unit variation. We will see how normalization and per-device calibration remove sensor-specific bias and scale so the model learns the signal rather than the serial number, and, crucially, how to fit those transforms inside the training fold only, so the cure for one leak does not quietly open another.