Time-aware cross-validation

"My model scored ninety-nine percent, and then I noticed I had been grading Monday's exam with Tuesday's answer key."

A Time-Traveling AI Agent

Why random folds leak time

Standard \(k\)-fold assigns each sample to a fold at random, then trains on \(k-1\) folds and tests on the one left out. That is exactly right when samples are exchangeable, which is what independent draws mean. Sensor streams are not exchangeable. Consecutive windows of an accelerometer, an ECG, or a vibration probe are heavily autocorrelated: the reading at time \(t\) is a near-copy of the reading at \(t-1\). When you shuffle, a test window at time \(t\) lands next to training windows at \(t-1\) and \(t+1\) that are almost identical to it. The model does not need to generalize; it can nearly memorize a neighbor and interpolate. This is temporal leakage, and it inflates every metric you report.

The damage has two distinct sources, and separating them tells you how to fix it. The first is window overlap: if you segment a stream into windows that slide by less than their length, a single physical instant appears in several windows, and shuffling can drop copies of one instant on both sides of the train/test line. The second is deeper and survives even non-overlapping windows: look-ahead. Any preprocessing fit on the whole dataset (a normalization mean, a PCA basis, a resampling filter, a label built from a future event) smuggles future statistics into the past. A model that learns the population mean has learned something it could not have known at inference time. Both problems reduce to the same violation: the evaluation lets information flow backward in time.

Forward chaining: expanding and sliding windows

The workhorse fix is forward-chaining cross-validation, also called walk-forward or rolling-origin evaluation. Sort everything by time, then cut the timeline into ordered blocks and evaluate on each block using only the blocks before it. Two variants matter. In the expanding-window scheme the training set grows: fold 1 trains on block 1 and tests on block 2, fold 2 trains on blocks 1 to 2 and tests on block 3, and so on. This mirrors a system that retrains on its full history. In the sliding-window scheme the training set is a fixed-length window that marches forward, discarding the oldest block each step. This mirrors a system that deliberately forgets stale data, which is the right model when the process drifts, a theme Chapter 66 takes up in full.

Which to choose is a modeling claim, not a coin flip. Use expanding windows when older data stays relevant, for example a gait classifier for a stable population. Use sliding windows when the world moves under you: a machine whose bearings degrade, a user whose behavior shifts with the seasons, or any streaming learner of the kind Chapter 60 deploys. Reporting the per-fold score across a forward chain also gives you something a single number cannot: a curve of performance over time that reveals whether the model is quietly decaying.

Purging and embargo: sealing the seam

Ordering the blocks is necessary but not sufficient. Two subtle leaks cross the train/test boundary even in a forward chain, and both are cured by leaving a gap. The first arises from overlapping windows and multi-step labels. If a training sample at time \(t\) carries a label that depends on the interval \([t, t+h]\) (say, "did a fault occur within the next \(h\) samples"), then its label overlaps the start of the test block. That training sample knows something about the test period. Purging removes any training sample whose label horizon reaches into the test set. The second leak is pure autocorrelation bleeding across the seam: the last training window and the first test window are adjacent in time and therefore nearly identical. An embargo discards a short stretch of samples immediately after the training block, so the test set begins only after the correlation has decayed.

How wide should the gap be? Set the purge to the label horizon \(h\), and set the embargo to roughly the autocorrelation length of the signal. A practical rule of thumb is

where \(W\) is the window length, \(s\) is the stride, and \(\tau\) is the number of samples over which windows still share raw data. When windows do not overlap (\(s \ge W\)) the second term vanishes and the embargo need only cover residual physical correlation. This purge-and-embargo discipline is the core of the combinatorial purged cross-validation proposed by Marcos Lopez de Prado for financial series, and it transfers cleanly to any autocorrelated sensor stream. The code below implements the essential idea, an expanding forward chain with a configurable embargo gap.

import numpy as np

def purged_walk_forward(n, n_splits=5, embargo=0):
    """Expanding-window CV: each train block precedes its test block,
    with an `embargo` gap of samples dropped between them."""
    block = n // (n_splits + 1)
    for k in range(1, n_splits + 1):
        train_end  = block * k
        test_start = train_end + embargo          # seal the seam
        test_end   = min(train_end + block, n)
        if test_start >= test_end:
            continue
        yield np.arange(0, train_end), np.arange(test_start, test_end)

for tr, te in purged_walk_forward(1000, n_splits=4, embargo=25):
    print(f"train 0..{tr[-1]:4d}   embargo 25   test {te[0]}..{te[-1]}")

As Listing 65.3 makes explicit, the training indices are always numerically below the test indices, so no future sample can reach the model, and the embargo carves out the neighbors that would otherwise leak by adjacency. Note also that time-aware splitting composes with the entity splitting of the previous section: to test generalization across both time and devices you keep each device's timeline intact and hold out later time and unseen devices together.

from sklearn.model_selection import TimeSeriesSplit
cv = TimeSeriesSplit(n_splits=5, gap=25)   # gap = embargo, no shuffling

Choosing and reporting the protocol

Three questions pin down the right design. What is the deployment cadence? If the fielded system retrains nightly on all history, evaluate with expanding windows; if it uses a fixed lookback, slide a matching window. What is the label horizon? Set the purge to it, or you leak through the labels. How fast does the signal decorrelate? Set the embargo to that timescale. When in doubt, prefer the more conservative choice: a slightly-too-wide gap costs you a little data, while a too-narrow gap costs you your credibility. Finally, always report the forward chain as a sequence of scores, not just their mean. A model that scores 0.9, 0.88, 0.7, 0.55 across four chronological folds is not a 0.76 model; it is a model that is falling apart, and only the time-ordered breakdown shows it. That decay is often the single most decision-relevant fact your evaluation produces, and it is exactly what feeds the drift monitoring of Chapter 69.

What's Next

In Section 65.4, we turn from when the test data lives to how rare the thing you care about is. Falls, machine failures, and arrhythmias are vanishingly infrequent, and the same accuracy that looks reassuring on a balanced split becomes a lie under heavy imbalance. We build the metrics and resampling protocols that keep rare-event evaluation honest.