Patient-level splits and leakage

"I scored 0.98 on the held-out set. It turns out I had simply learned to recognize Mr. Patel's heartbeat, and Mr. Patel was in both piles."

An Overfitted AI Agent

Why a random split lies to you

The default in most tutorials is a random split: shuffle every window, deal 80 percent to train and 20 percent to test. For independent samples this is fine. Biosignal windows are not independent. Consecutive heartbeats from one Holter recording share the same lead placement, the same electrode-skin impedance, the same QRS morphology, the same baseline wander, and the same idiosyncratic physiology. Two windows one second apart are near-duplicates. A random split scatters these near-duplicates across train and test, so the test set is answered largely by lookup rather than by generalization.

The consequence is a systematic optimistic bias. Formally, honest generalization asks how a model trained on patients \(P_{\text{train}}\) performs on a new patient \(p^\ast \notin P_{\text{train}}\). A random split instead estimates performance conditioned on having already seen \(p^\ast\), a quantity no deployment will ever enjoy. The gap between the two can be enormous. On arrhythmia and seizure benchmarks, moving from record-mixed to subject-disjoint evaluation routinely erases ten to thirty points of reported accuracy. The model was never that good; the protocol was that generous.

The taxonomy of biosignal leakage

Patient overlap is the headline offender, but several quieter leaks share the same signature: a validation score that will not survive contact with a new hospital. Naming them makes them checkable.

Identity leakage is the same person in two splits, the case above. Recording leakage is subtler: even with disjoint patients, overlapping sliding windows from a single recording can straddle a fold boundary, so a window centered at \(t\) and another at \(t+1\) share most of their samples. Enforce a guard gap so no two windows within a stride of the boundary are separated. Device and site leakage is the trap where a rare class was collected on one machine or at one clinic: the model learns the device's transfer function, not the disease, a failure mode we return to under distribution shift in Chapter 66. Preprocessing leakage happens when you fit a normalizer, a filter's statistics, or a PCA basis on the full dataset before splitting, letting test-set statistics bleed into training; fit every transform on the training fold only. Label leakage occurs when the reference standard (Section 28.5) encodes a shortcut, for example a lead that was only connected when the annotating cardiologist already suspected the finding.

Grouped cross-validation, and its stratified cousin

The mechanism that prevents identity leakage is grouped cross-validation: attach a group label (the patient ID) to every sample and require that no group crosses a fold boundary. In scikit-learn this is GroupKFold. The complication is that clinical datasets are also severely imbalanced: a positive class may live in a handful of patients, and a naive grouped split can hand an entire fold zero positives, making its metric meaningless. StratifiedGroupKFold solves both at once, keeping groups intact and balancing class proportions across folds as far as the group structure allows.

import numpy as np
from sklearn.model_selection import StratifiedKFold, StratifiedGroupKFold
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(0)
n_patients, beats = 60, 40
# Each patient has an identity signature; label depends only on a weak clinical feature.
pid   = np.repeat(np.arange(n_patients), beats)
ident = rng.normal(size=n_patients)[pid]                 # strong per-person shortcut
y     = np.repeat(rng.integers(0, 2, n_patients), beats) # label is a patient property
clin  = 0.4 * y + rng.normal(scale=1.0, size=len(y))     # weak honest signal
X     = np.column_stack([clin, ident + rng.normal(scale=0.05, size=len(y))])

def cv_auc(splitter, groups=None):
    scores = []
    for tr, te in splitter.split(X, y, groups):
        m = LogisticRegression().fit(X[tr], y[tr])
        scores.append(roc_auc_score(y[te], m.predict_proba(X[te])[:, 1]))
    return np.mean(scores)

naive   = cv_auc(StratifiedKFold(5, shuffle=True, random_state=0))
grouped = cv_auc(StratifiedGroupKFold(5), groups=pid)
print(f"random (leaky) AUC : {naive:.3f}")
print(f"patient-disjoint AUC: {grouped:.3f}")

Run this and the two numbers disagree sharply, and the disagreement is the entire lesson: the gap between them measures exactly how much your reported score depends on having seen the patient before. Report both during development and treat the gap as a leakage alarm, not a nuisance.

Nested groups, and when the person is not enough

Sometimes the person is the wrong grouping key because a coarser correlation dominates. If each hospital used a different ECG cart, or each study site enrolled a different demographic, then even patient-disjoint folds can leak site identity. The principle generalizes: split on the coarsest unit that shares a nuisance signature. For a multi-site trial that often means leave-one-site-out evaluation nested inside patient-disjoint folds, which simultaneously stress-tests generalization to new hardware and new populations. This hierarchical view (person within device within site) connects to the broader benchmarking protocol developed in Chapter 65.

A related caution belongs to privacy, not just accuracy. Because a physiological signal re-identifies its owner, a model that has memorized patients can leak who they are, an attack surface that Chapter 34 examines under biometric re-identification. Patient-disjoint splitting is thus doing double duty: it makes your metrics honest and it discourages the model from encoding identity in the first place.

What's Next

In Section 28.7, we widen the lens from evaluation hygiene to the regulatory and safety context that surrounds any biosignal model destined for a real body, previewing the standards, risk classes, and clinical-validation expectations that later chapters in this part make concrete.