Sensor windows as tensors; channel and axis structure

"They handed me a tensor and forgot to say which axis was time. I convolved across my sensors and learned the shape of a wiring harness."

A Disoriented AI Agent

From a continuous stream to a fixed-shape window

A raw sensor produces an unbounded, ragged thing: samples arriving over time, possibly at different rates per modality, with gaps and jitter. A neural network wants the opposite: a batch of fixed-shape tensors it can stack and matrix-multiply. Bridging the two is windowing, and it is the first modeling act of the chapter. You slice the stream into segments of length \(L\) samples, advancing by a stride \(S\) between segments, and treat each segment as one training example. What you produce is a tensor; why you window is that convolutions and recurrences need a bounded, aligned tensor to operate on; how you choose \(L\) and \(S\) is a physics-and-labels decision, not a default.

Window length \(L\) sets the receptive horizon: it must be long enough to contain the phenomenon you want to detect. A gait cycle at \(50\ \mathrm{Hz}\) runs about one second, so a window shorter than \(L=50\) samples cannot contain a full stride and no filter downstream can recover what the window threw away. Stride \(S\) sets overlap: \(S=L\) gives disjoint windows, \(S \[ N = \left\lfloor \frac{T - L}{S} \right\rfloor + 1. \]

Overlap is tempting because it inflates the dataset for free, but that free lunch is exactly where leakage hides: if adjacent overlapping windows land on opposite sides of a train/test split, the test set contains samples the model already saw. The window is the atomic unit that must respect subject-level and time-level splits, a rule Chapter 5 makes non-negotiable.

The three axes: batch, channel, time

Once windowed, a batch of sensor examples is a rank-3 tensor. Its three axes are batch \(N\) (independent windows stacked for parallel processing), channel \(C\) (the simultaneous scalar streams the device emits), and time \(L\) (samples within one window). For a nine-axis inertial measurement unit, \(C=9\): three accelerometer axes, three gyroscope axes, three magnetometer axes, each a separate row that shares the same clock. The channel axis is where a multivariate sensor differs from a grayscale image: an image has one intensity channel over two spatial axes, while a sensor window has many physical channels over one temporal axis.

Why does the channel axis deserve its own name rather than being flattened into time? Because a convolution treats the two axes with opposite symmetry. Along time a 1D convolution is translation-equivariant: a pattern shifted later in the window produces the same response shifted later, which is exactly the invariance you want for a gait event that can occur anywhere in the window. Along channels there is no such symmetry: axis \(x\) of the accelerometer is not a shifted copy of axis \(y\), and a filter is free to weight the nine channels differently at every time step. Convolution slides over time and mixes over channels. Flatten the channel axis into time and you destroy that distinction, inviting the network to convolve one accelerometer axis into the next as though they were consecutive instants. The when to worry is any moment your tensor's axis order is ambiguous, which is most of the time, because frameworks disagree.

The framework conventions are worth memorizing because they are inconsistent by design. PyTorch's nn.Conv1d expects channels-first, \((N, C, L)\), so the sensor-channel axis sits in the middle. Keras and TensorFlow's Conv1D default to channels-last, \((N, L, C)\), so time sits in the middle. NumPy loaders and pandas frames usually hand you \((L, C)\) per window, matching how a CSV reads (rows are time, columns are channels), which happens to align with Keras and needs a transpose for PyTorch. None of these is more correct; they are different contracts, and the entire job here is knowing which contract the next layer expects and transposing exactly once, deliberately, with a comment.

Building the tensor in code

Windowing is the kind of loop everyone writes once, gets subtly wrong at the edges, and then never trusts again. The code below builds windows two ways: an explicit loop that makes the arithmetic visible, and a strided view that a library computes without copying. The explicit version documents intent; the strided version is what you ship.

import numpy as np

def window_stream(stream, L, S):
    """stream: (T, C) array from a CSV loader. Returns (N, C, L), PyTorch channels-first."""
    T, C = stream.shape
    N = (T - L) // S + 1
    out = np.empty((N, C, L), dtype=stream.dtype)
    for i in range(N):
        seg = stream[i * S : i * S + L]      # (L, C): time-major, as CSV reads
        out[i] = seg.T                        # -> (C, L): channels-first for Conv1d
    return out

# A 10-second, 3-axis accelerometer recording at 50 Hz
rng = np.random.default_rng(0)
stream = rng.standard_normal((500, 3)).astype(np.float32)   # (T=500, C=3)
X = window_stream(stream, L=128, S=64)
print(X.shape)                    # (6, 3, 128) = (N, C, L)

# The zero-copy library equivalent (NumPy >= 1.20)
from numpy.lib.stride_tricks import sliding_window_view
V = sliding_window_view(stream, window_shape=128, axis=0)[::64]  # (6, 3, 128)
print(V.shape, np.allclose(V, X)) # (6, 3, 128) True

As Listing 13.1 shows, the strided view returns the same \((N, C, L)\) tensor the loop builds, but it does so without allocating \(N \times C \times L\) fresh floats: overlapping windows share their underlying bytes. That matters at scale, where a \(50\%\)-overlap dataset would otherwise double your memory footprint for numbers that are literally the same samples viewed twice.

Channels are not interchangeable: physical structure on the channel axis

The channel axis carries more than a count; it carries physical grouping the model can exploit or ignore. Nine IMU channels are really three modalities of three axes each, and those groupings are not arbitrary permutations. The three accelerometer axes form an orthonormal frame, so rotating the device permutes and mixes them in a structured way, which is why augmentation in Section 13.5 can rotate the accelerometer triple as a rigid body but must never shuffle an accelerometer axis with a magnetometer axis. Channels can also differ in units, scale, and noise floor: an accelerometer in \(\mathrm{m/s^2}\), a gyroscope in \(\mathrm{rad/s}\), a barometer in \(\mathrm{hPa}\). A convolution that mixes them with shared-scale weights will be dominated by whichever channel has the largest raw magnitude until per-channel normalization fixes it, another Section 13.5 topic.

This is also where the two dominant design philosophies of the rest of Part IV first appear. A channel-mixing model lets the first layer combine all \(C\) channels at every time step, learning cross-sensor interactions (accelerometer and gyroscope together disambiguate rotation from translation). A channel-independent model processes each channel with its own filters and fuses them only later, which is more robust when channels can drop out or arrive at different rates. Both are legitimate; the choice is an inductive bias you set through the tensor's grouping, revisited for transformers in Chapter 15. What you cannot do is pretend the channel axis is featureless. Where a feature vector from Chapter 8 discards temporal order to summarize a window, the tensor here preserves both order along time and identity along channels, and that preserved structure is the entire reason a convolutional network can outperform a bag of hand-crafted features.

What's Next

In Section 13.2, we put the first learnable layer on top of the \((N, C, L)\) tensor: the 1D convolution that slides along the time axis, mixes across the channel axis exactly as this section anticipated, and the temporal pooling that shrinks the time dimension into a decision. The axis discipline you just fixed is what makes those filters mean anything.