Inductive biases for time series

"Give me enough data and I will learn anything. Give me a good prior and I will need far less data, provided you were right about the world."

A Well-Regularized AI Agent

What an inductive bias is, and why sensor time series need a strong one

A model that could represent every possible function fits your training set perfectly and predicts nothing useful, because infinitely many functions agree on the points you saw and disagree everywhere else. Learning is only possible because we constrain the hypothesis space: we tell the model, before it sees any data, which functions are a priori plausible. That constraint is the inductive bias. It is the reason a network trained on Monday's walking data recognizes Tuesday's walking data at all.

The why is a sample-efficiency argument. A strong, correct bias shrinks the effective number of parameters the data must pin down, so you need less labeled data to reach a given accuracy. This matters more for sensors than for almost any other domain, because sensor labels are expensive: a human activity dataset might have millions of accelerometer samples but only a few thousand labeled activity segments, and every label costs annotator time or a supervised protocol. A model with the right bias spends that scarce supervision learning what is genuinely uncertain instead of rediscovering that time flows left to right. The flip side, the when it hurts, is unforgiving: a strong bias that is wrong for your signal caps your accuracy no matter how much data you add, because the true function was excluded from the hypothesis space before training began.

A catalog of temporal inductive biases

Deep models for sensor streams encode a small, recurring set of assumptions. Naming them lets you audit an architecture the way you would audit a set of modeling premises.

Shift equivariance and shift invariance. A convolution shares one filter across every time step, which means a pattern shifted in time produces the same response shifted in time (equivariance); add a global pooling layer and the final representation stops caring where the pattern occurred at all (invariance). The assumption is that the identity of an event does not depend on its absolute position in the window. A footstep is a footstep whether it lands at sample 10 or sample 90.

Locality. A finite kernel of width \(k\) asserts that what a sample means is determined by its immediate neighborhood, not by samples a thousand steps away. Formally the receptive field bounds the interactions the layer can model, so locality is the bet that the signal's short-range structure carries most of the information. Dilation and depth relax this bound gradually, which is precisely the multiresolution story of Section 13.3.

Causality. A causal model computes the output at time \(t\) from inputs at times \(\le t\) only, never peeking at the future. This is not a mathematical nicety; it is a hard requirement for any system that must act while the stream is still arriving, and it shapes every streaming deployment in Section 14.5. Offline analysis can drop the constraint and see the whole window at once.

Channel symmetry. How you wire the sensor axes encodes an assumption about their relationship. A channel-independent model applies the same transform to each axis and asserts the axes are exchangeable; a channel-mixing model lets them interact and asserts their joint pattern matters. For a tri-axial accelerometer the axes are emphatically not exchangeable once gravity picks out a direction, a subtlety we return to below.

Smoothness and periodicity. Many physical signals are band-limited and locally smooth, so architectures that favor low-frequency, continuous responses match reality; others, like gait or the cardiac cycle, are quasi-periodic, and encoding that repetition (through pooling over cycles or explicit periodic structure) is a powerful prior. These continuity assumptions connect directly to the filtering view in Chapter 6.

You can watch shift invariance appear or fail to appear just by changing how the time axis is collapsed. The snippet below feeds a three-axis window and a time-shifted copy of it through two encoders and measures how much the output moves.

import torch, torch.nn as nn

torch.manual_seed(0)
x = torch.randn(1, 3, 128)                      # (batch, channels, time): a 3-axis IMU window
shifted = torch.roll(x, shifts=16, dims=-1)     # slide the same window 16 samples in time

# Conv + global average pool: shift invariance is baked into the architecture
conv = nn.Sequential(
    nn.Conv1d(3, 8, kernel_size=9, padding=4), nn.ReLU(),
    nn.AdaptiveAvgPool1d(1), nn.Flatten())      # collapse the time axis to one vector

# Flatten + linear: every time position gets its own weights, so no shift bias
mlp = nn.Sequential(nn.Flatten(), nn.Linear(3 * 128, 8))

for name, net in [("conv+pool", conv), ("flatten+linear", mlp)]:
    delta = (net(x) - net(shifted)).abs().max().item()
    print(f"{name:15s} max output change after a 16-sample shift: {delta:.4f}")

As Listing 13.6.1 makes concrete, "shift invariance" is a property of the wiring, not something the optimizer has to discover from data. Choosing the conv encoder is choosing the assumption before a single gradient step.

from pytorch_tcn import TCN
model = TCN(num_inputs=3, num_channels=[16, 16, 32], causal=True)  # causal=True does the masking

Matching the bias to the physics

The catalog is useless without judgment about when each assumption holds. The discipline is to read the bias back as a claim about the sensor and ask whether the claim is true.

Shift invariance is the classic example of a bias that is right until it is not. For classifying a gesture inside a fixed window it is exactly right. But a tri-axial accelerometer also measures the gravity vector, and gravity encodes orientation: the same motion performed with the phone upside down produces a differently signed signal. A model that is invariant to absolute channel orientation will happily confuse the two. The fix is either to remove the assumption (let the axes mix and keep orientation) or to remove the nuisance first (rotate into a gravity-aligned frame), a design tension explored in depth in Chapter 23. Stationarity is another quiet premise: weight sharing across time assumes the signal's statistics do not change within the window, which holds for a two-second walking clip and fails for a startup transient or a fault that evolves over minutes.

The bias-strength spectrum: from hand-crafted priors to data-hungry generalists

Architectures for sensor streams sit on a spectrum of how much structure they impose. At the strong end are hand-crafted feature pipelines (Chapter 8): a spectral-band energy feature is a maximal prior, encoding exactly what to look at, and it can be trained on a few hundred examples. Convolutional and temporal-convolutional models sit in the middle, imposing locality, weight sharing, and (optionally) causality while learning the filters themselves. At the weak end, attention-based models (Chapter 15) drop locality entirely and let any time step attend to any other, buying flexibility at the cost of needing far more data or clever pretraining to avoid overfitting.

The trend toward foundation models is, in one sentence, the trade of inductive bias for data. When you cannot supply the right prior by hand, you can sometimes supply enough unlabeled data that the model learns an appropriate bias itself, which is exactly the promise of the time-series foundation models in Chapter 19. This is not a free win. Data at that scale is its own expense, and a learned bias is harder to audit than a stated one. For most sensor projects, with modest labeled data and a well-understood signal, a mid-spectrum model carrying the few biases you can justify remains the sample-efficient default. The classical-versus-foundation tradeoff is a recurring thread of this book, and inductive bias is the axis it turns on.

What's Next

In Section 13.7, we stop reasoning about biases in the abstract and open the model up: how to visualize and inspect the filters a network actually learned, so you can confirm that the assumptions you thought you baked in are the ones the trained weights are honoring, and catch the cases where the optimizer quietly built a shortcut instead.