Streaming/stateful inference

"They trained me on the whole recording at once, then deployed me to watch the sensor one sample at a time, forever, with no rewind button. I would like a word with whoever thought those were the same job."

A Streaming AI Agent

This section assumes the recurrent cells of Section 14.1 and the causal dilated convolutions of Section 14.3, and it inherits the streaming-safe conditioning rules from Chapter 13. The timing vocabulary (sample period, latency, jitter) is the one established in Chapter 3. Here we care about one question: given a model that was trained to consume a whole window, how do we run it on an endless stream without either recomputing everything or changing the answer?

Two machines: offline windows versus an endless stream

Offline, you control the tensor. You pad it, you see both edges, you can even look ahead. Online, the future does not exist yet, so any layer that peeks forward is unusable and the model must be causal: the output at time \(t\) depends only on samples up to \(t\). Both architectures in this chapter can be made causal (RNNs are causal by construction; a TCN becomes causal with left-only padding), which is exactly why they stream and a bidirectional model does not. The second difference is memory. An offline forward pass allocates the full \(T\)-length activation maps; a streaming pass may keep only what the next step needs. The whole discipline reduces to identifying that minimal carried quantity, the state, and updating it in constant time.

Streaming an RNN: the state was there all along

Recurrent networks are the easy case, because recurrence is streaming. An LSTM or GRU already computes its answer through a per-step update

where \(h_t\) (for an LSTM, the hidden and cell pair \((h_t, c_t)\)) is a fixed-size vector independent of how long the stream has run. To go from batch training to streaming inference you do not touch the weights; you simply persist \(h_t\) across calls instead of re-initializing it to zero for each window. The per-sample cost is one cell update, \(O(d^2)\) in the hidden size \(d\) and, crucially, constant in stream length. The only new decisions are administrative: when does a session start (reset \(h\) to zero), and what happens when the stream has a gap.

Streaming a TCN: cache the receptive field, do not recompute it

Temporal convolutional networks are the case where naive streaming quietly wastes almost everything. A TCN's output at time \(t\) depends on the last \(R\) samples, where \(R\) is the receptive field set by the dilation schedule of Section 14.3. The tempting deployment is: keep a buffer of the last \(R\) samples, and on every new sample re-run the full stack over that buffer. That works and costs

summed over layers \(\ell\) with kernel size \(K_\ell\) and channel count \(C_\ell\). But the convolution at layer \(\ell\) for all but the newest position was already computed on the previous sample and has not changed. The fix is a per-layer ring buffer that caches exactly the dilated taps each layer needs. Each layer then performs a single kernel evaluation per new sample, and the cost drops to

a factor of \(R\) less work, with \(R\) often in the hundreds. The state you carry is one small first-in-first-out buffer per layer, sized to that layer's dilation times its kernel, and the total memory is proportional to the receptive field rather than to the (infinite) stream. The parallel with a Kalman filter's carried state (Chapter 9) is not a coincidence: both replace an unbounded history with a fixed-size sufficient summary.

import torch, torch.nn.functional as F

class StreamingCausalConv1d:
    """Causal dilated conv that consumes ONE sample per call, O(1) in stream length."""
    def __init__(self, conv):                       # conv: a trained nn.Conv1d
        self.conv = conv.eval()
        self.k, self.d = conv.kernel_size[0], conv.dilation[0]
        span = (self.k - 1) * self.d + 1            # samples the kernel spans
        self.buf = torch.zeros(1, conv.in_channels, span)  # ring of the receptive span

    def step(self, x_t):                            # x_t: (1, in_channels)
        self.buf = torch.roll(self.buf, -1, dims=2)
        self.buf[:, :, -1] = x_t                    # newest sample at the right edge
        taps = self.buf[:, :, ::self.d]             # pick the dilated taps only
        with torch.no_grad():
            return self.conv(taps)[:, :, -1]        # (1, out_channels): output for THIS t

The code above is the whole idea in miniature: state is the buffer, the update is a shift-and-fill, and the output is a single column rather than a recomputed window. Chain one StreamingCausalConv1d per TCN layer and the stack streams end to end. One subtlety survives from Chapter 13: any BatchNorm must run on frozen inference statistics, never on a per-window batch mean, or your streaming output will not match the offline one and your parity test will (correctly) fail.

from pytorch_tcn import TCN
net = TCN(num_inputs=1, num_channels=[16]*8, kernel_size=3, causal=True)
net.eval(); net.reset_buffers()          # zero the streaming state at session start
for x_t in stream:                       # x_t: (1, 1, 1), one sample at a time
    y_t = net.inference(x_t)             # buffers updated in place, O(1) per step

Warm-up, resets, and chunking: the operational details

Three practical questions decide whether a streaming model behaves in the field. First, warm-up: at session start the state is zero and the buffers are empty, so the first \(R\) samples (or, for an RNN, the first several dozen steps) run on partial context and are less reliable. Either mark those outputs as low-confidence or prime the model with a short lead-in before you trust it. Second, reset semantics: state carried across a genuine discontinuity is worse than useless. When a new recording begins, a wearer removes the device, or a data gap breaks temporal continuity (Section 14.6 treats irregular timing in depth), reset the state so stale history does not bleed into a fresh context. Third, chunking: strict one-sample-at-a-time streaming minimizes latency but maximizes per-call overhead. Processing short blocks of, say, 16 or 64 samples amortizes Python and kernel-launch overhead and often improves throughput on an accelerator, at the cost of adding one block of latency. The right chunk size is a latency-versus-throughput dial set by your deployment target, a tradeoff that the edge-deployment treatment of Chapter 60 makes central.

What's Next

In Section 14.6, we drop the comfortable assumption that samples arrive on a fixed clock. Real sensor streams are irregular and asynchronous: channels report at different rates, packets are late or missing, and the gap between samples carries information. We will see how to make recurrent and convolutional models consume time itself as an input rather than pretending every step is evenly spaced, which is exactly the discontinuity that forced the state resets above.