Segmentation and transition states

"I was ninety-eight percent sure the user was sitting and ninety-eight percent sure they were standing. The half second in between, when they were neither, is the only part anyone ever asks me about."

An Over-Confident AI Agent

Why the continuous stream resists cutting

The dominant approach is the sliding window: slice the stream into fixed-length frames of length \(W\) with hop \(H\), extract features from each, and classify each frame independently. It is simple, streams naturally, and dominates the literature. It also hides an unavoidable tension in the choice of \(W\). A window must be long enough to contain the periodic structure of the activity: a full gait cycle for walking is roughly one second, so a \(0.5\ \mathrm{s}\) window literally cannot see a stride and will confuse walking with fidgeting. But a long window is more likely to straddle a boundary and contain two activities at once, which no single label describes. This is the segmentation dilemma stated as a bias-variance style tradeoff: short windows resolve boundaries sharply but lack the temporal context to classify well, long windows classify robustly but smear every transition.

Formally, with a hop \(H\) the fraction of windows that are "mixed" (contain a true boundary) grows with \(W\). If activity segments have mean duration \(\bar{d}\), the expected number of boundary-straddling windows per segment is about \(W/H\), independent of \(\bar{d}\), so short activities suffer most: a two-second bout of stair-climbing sampled with a three-second window may never produce a single pure window. Overlapping windows (small \(H\)) improve temporal resolution and give more training examples, but they also make adjacent windows highly correlated, which quietly inflates any accuracy number computed by splitting windows randomly rather than by subject or by time.

Two families of segmentation

Segmentation strategies split into two families. Explicit (bottom-up) segmentation first finds boundaries, then classifies the resulting variable-length segments. Boundaries come from change-point detection on the signal statistics: a sit-to-stand event shows up as a sharp change in the mean and variance of vertical acceleration, exactly the kind of shift the CUSUM and sliding-window change detectors of Chapter 12 are designed to catch. The appeal is that each classified segment is internally homogeneous, so the classifier never sees a mixed window. The risk is cascading error: a missed boundary merges two activities forever, and an early boundary detector run on raw acceleration will fire on every footstep unless you smooth or threshold it carefully.

The listing below applies an off-the-shelf change-point search to a synthetic accelerometer-magnitude trace with three regimes (still, walking, still) and recovers the two boundaries.

import numpy as np
import ruptures as rpt

fs = 50.0
t = np.arange(0, 12, 1 / fs)
sig = np.full(t.size, 9.81)                     # accel magnitude, resting
walk = (t > 4) & (t < 8)                        # a walking bout
sig[walk] += 2.5 * np.sin(2 * np.pi * 1.8 * t[walk])   # ~1.8 Hz gait
sig += np.random.normal(0, 0.15, t.size)

# Pelt search on a shift in mean+variance; penalty controls sensitivity.
algo = rpt.Pelt(model="rbf", min_size=int(1.0 * fs)).fit(sig)
bkps = algo.predict(pen=12.0)                   # indices of change points
print("boundaries at t =", [round(b / fs, 2) for b in bkps[:-1]], "s")

As Listing 26.3 shows, boundary-finding can be a few lines rather than a hand-rolled CUSUM. The second family, implicit (dense) segmentation, skips explicit boundaries entirely: a sequence model emits a label at every timestep, and boundaries simply appear wherever the predicted label changes. Recurrent and temporal-convolutional networks from Chapter 14 do this naturally, learning transition dynamics from data instead of from a hand-tuned detector. This is now the default for high-quality systems because it makes the boundary a learned, context-aware decision rather than a threshold on a single statistic. Its cost is data hunger and the need for densely labelled streams, which are far more expensive to annotate than isolated clips.

Evaluating a segmentation without fooling yourself

Segmentation quality cannot be judged by frame accuracy alone, and here the running leakage theme from Chapter 65 bites twice. First, frame-level accuracy is dominated by the long, easy, stationary spans and is nearly blind to the short transitions you most care about; a model can score 94% frame accuracy while missing half of all sit-to-stand events. Report event-level metrics too: count a predicted segment as a true positive only if it overlaps the correct ground-truth segment by more than some threshold, and report insertions, deletions, merges, and fragmentation, the classic segmentation error taxonomy. Second, because overlapping windows are strongly correlated in time, splitting them randomly into train and test leaks near-identical neighbours across the split and inflates every number. Always split by subject and by contiguous time block, never by window.

A final, subtler point concerns the boundaries themselves: human annotators disagree about exactly where a transition starts by a few hundred milliseconds, so demanding frame-exact boundary agreement penalizes a model for beating the noise floor of the labels. Score boundaries with a tolerance window (a predicted boundary within, say, \(\pm 0.5\ \mathrm{s}\) of a labelled one counts as correct), matching the intrinsic imprecision of the ground truth rather than pretending it is exact.

What's Next

In Section 26.4, we turn from where to cut the stream to whose stream it is. Segmentation and classification both degrade when a model trained on one population meets a new user whose gait, pace, and transition style differ, so we take up user personalization and adaptation: how to bend a general model toward the individual wearing it without collecting a fresh labelled dataset for every person.