Windowed and stateful inference

"The batch version of me got to see the whole song before naming the tune. Now I have to name it while it is still being played, and I am not allowed to forget the opening notes."

A Real-Time AI Agent

From an endless stream to finite windows

A trained sensor model almost always expects an input of fixed shape: a tensor of \(W\) timesteps by \(C\) channels. The stream, by contrast, is unbounded. The job of windowing is to reconcile the two by defining a moving frame over the stream. Three parameters describe any windowing scheme. The window length \(W\) is how many samples the model sees at once, the what of the receptive field. The hop \(H\) (also called the stride or step) is how far the frame advances between successive inferences. The overlap is the remainder \(W - H\), the number of samples two consecutive windows share.

These three choices name the classical taxonomy. When \(H = W\) there is no overlap and the frames tile the stream edge to edge: these are tumbling windows, and every sample is scored exactly once. When \(H < W\) the frames overlap and slide forward by less than their length: these are sliding or hopping windows, and each sample participates in \(W/H\) inferences. The degenerate case \(H = 1\) advances one sample at a time and is a true sample-by-sample sliding window, maximally responsive and maximally expensive.

Why does the hop matter so much? Because it sets your decision latency and your compute rate simultaneously, and they pull in opposite directions. A model cannot emit its first answer until a full window has filled, so at sample rate \(f_s\) the startup latency is \(W/f_s\). After that, a fresh answer arrives every \(H/f_s\) seconds, so the inference rate is \(f_s/H\) per second. Shrinking \(H\) makes the system more responsive and produces smoother output, but multiplies how often the model runs. Enlarging \(W\) gives the model more context (better for slow phenomena like a gait cycle or a bearing fault tone) but delays the first decision and blurs the boundary of fast events. The window length is a bet on the timescale of what you are trying to detect.

Mechanically, windowing is implemented with a ring buffer (circular buffer): a fixed array of length \(W\) into which new samples are written, overwriting the oldest, with a running index. When the buffer has advanced by \(H\) new samples since the last inference, you snapshot the current window and score it. This is \(O(1)\) memory per channel regardless of stream length, which is exactly why it works on a device with kilobytes of RAM. The snippet below turns a sample stream into windows without ever allocating in the hot path.

from collections import deque

def windowed(stream, W=128, H=32):
    """Yield overlapping windows of length W, advancing H samples each step."""
    buf = deque(maxlen=W)          # ring buffer: O(W) memory, drops oldest
    since_last = 0
    for sample in stream:          # sample: one (C,) vector per timestep
        buf.append(sample)
        since_last += 1
        if len(buf) == W and since_last >= H:
            since_last = 0
            yield list(buf)        # snapshot the current window -> model input

# 50 Hz triaxial accelerometer, 2.56 s window, 0.64 s hop (75% overlap)
for window in windowed(accel_stream, W=128, H=32):
    prob_fall = model.predict(window)

Listing 60.1.1 makes the ring-buffer discipline explicit: the memory footprint is set by \(W\) alone, and the since_last >= H guard is what converts a per-sample loop into a per-window inference cadence.

Stateless windowing versus stateful streaming

Windowing as described so far is stateless: each window is a self-contained input, scored with no memory of the windows before it. For a convolutional or feedforward model on overlapping windows this is simple and robust, but it hides a quiet inefficiency. With \(75\%\) overlap, three quarters of every window is recomputed from scratch on the next step: the model re-reads history it already processed one hop ago. For a small classifier that waste is negligible. For a deep model or a long window it can dominate the compute bill.

Stateful streaming attacks that waste head on. Recurrent models (Chapter 14) and modern state-space models like Mamba and S4 (Chapter 16) summarize everything they have seen into a fixed-size hidden state \(h_t\). The recurrence

means the model can consume the stream one sample (or one small chunk) at a time, updating \(h_t\) in place, and never re-reads the past because the past already lives in the state. Instead of feeding a \(W\)-sample window every hop, you feed the \(H\) genuinely new samples and let the carried state supply all prior context. The per-step cost drops from \(O(W)\) to \(O(H)\), and in the sample-by-sample limit to \(O(1)\), independent of how much history matters. A Kalman filter (Chapter 9) is the classical ancestor of exactly this idea: its state carries the entire relevant past forward so each new measurement is cheap.

That management burden is real and is where stateful deployments go wrong. Three hazards recur. First, state initialization: the very first chunk of a stream has no valid history, so \(h_0\) must be a sensible default (usually zeros or a learned initial state), and the first few outputs are warm-up transients that should be trusted less. Second, state reset at boundaries: when one logical stream ends and another begins (a new patient, a new machine, a new recording session, or a gap after dropped data), the carried state from the old context must be cleared, or the model will hallucinate continuity across a discontinuity that never existed. Third, train/serve consistency: a model trained on independent windows but served with carried state, or vice versa, sees a distribution it was never fit on. You must stream at inference the same way you streamed during training, and you must reset on the same boundaries, or the leakage-safe evaluation you did offline (Chapter 5) no longer predicts online behaviour.

Sizing the window and honoring the deadline

Putting the pieces together, a streaming inference design is fixed by four numbers and one policy. The four numbers are the sample rate \(f_s\), the window length \(W\), the hop \(H\), and the per-inference model latency \(\ell\). The policy is stateless-versus-stateful. From these you can predict everything an operator cares about before deployment. The worst-case time from an event to its first possible detection is the fill-plus-hop latency \((W + H)/f_s\), and the end-to-end latency to a delivered decision adds the model's own \(\ell\). The sustained compute load is one inference every \(H/f_s\) seconds, so the model must satisfy \(\ell < H/f_s\) or windows pile up faster than they clear, the failure mode that spills into the backpressure and data-loss problems of Section 60.2. The memory footprint is the ring buffer (\(W \times C\) for stateless) or the hidden state (fixed, independent of \(W\), for stateful).

These constraints frequently conflict, and resolving the conflict is the design. If the deadline demands a small \((W+H)/f_s\) but the phenomenon needs a large \(W\) to be recognizable, stateless windowing cannot satisfy both, and a stateful model that keeps long context in \(h_t\) while consuming small hops is often the only escape. If the compute budget cannot sustain \(f_s/H\) inferences with the required \(W\), overlap-heavy stateless scoring is off the table and statefulness again earns its keep. The window is where the timescale of the world meets the timescale of your hardware, and the two rarely agree without negotiation.

from river.utils import Rolling
from river import stats

# a rolling mean over the last 128 samples, updated per sample in O(1)
roll = Rolling(stats.Mean(), window_size=128)
for x in accel_stream:
    roll.update(x)
    feature = roll.get()   # always reflects the current window

What's Next

In Section 60.2, we confront what happens when the arithmetic of this section fails: when samples arrive faster than the model can clear them and the ring buffer starts to overflow. We take up backpressure, load shedding, and principled data loss, the discipline of deciding which data to drop and how to keep a streaming system honest when it cannot keep up.