Attention over time and channels

"You asked me to summarize the last ten minutes of vibration. I looked at all of it at once and decided which forty milliseconds actually mattered."

An Attentive AI Agent

Scaled dot-product attention: a content-addressed average

Attention takes a set of tokens, here a set of vectors extracted from the sensor stream, and rebuilds each one as a weighted blend of all the others. Every token is projected three ways by learned matrices into a query \(q\), a key \(k\), and a value \(v\). The query says what this token is looking for; the key says what each token offers; the value is the content it contributes if selected. Stacking the tokens of one window into matrices \(Q\), \(K\), and \(V\), the whole operation is:

$$\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V$$

Read it inside out. The product \(QK^{\top}\) scores every query against every key by dot product, producing an \(n \times n\) grid of compatibilities for a window of \(n\) tokens. Dividing by \(\sqrt{d_k}\), the square root of the key dimension, keeps those scores from growing with dimension and saturating the softmax into a near one-hot spike; this is the scaled in the name and it matters for stable gradients. The softmax turns each row into a probability distribution, the attention weights, and multiplying by \(V\) forms, for each token, a convex combination of all values weighted by relevance. The result is content addressing: a token retrieves information not by position but by what it resembles. In practice we compute several such maps in parallel with different projections, multi-head attention, so one head can track a slow trend while another locks onto transients, and their outputs are concatenated and mixed.

Attention over time: which moments explain each other

Point the mechanism along the time axis and each token is one time step (or, after Section 15.2, one short patch of time). The attention weights then form a map from each moment to every other moment, saying which instants the model consults to represent the present one. This is a strict generalization of what the temporal models of the previous chapter did. Where a causal convolution averages a fixed neighborhood with fixed weights, temporal attention averages the whole window with weights that depend on the signal, so a heartbeat can attend to the previous beat to measure an interval, and a gait classifier can line up the current stride against the one before it even if the cadence drifted. Because the raw operation is permutation invariant, it has no built-in sense of order; the model only learns that step \(t{-}1\) precedes step \(t\) once you add the position information of Section 15.3. For streaming and forecasting you also apply a causal mask that sets the scores for future positions to \(-\infty\) before the softmax, so a token can attend only to the past, preserving the real-time constraint that disqualified bidirectional recurrence in Chapter 14.

The listing below runs one head of self-attention over a short multichannel window and reads off the temporal attention map, the row of weights that says where the model looks.

import torch, torch.nn.functional as F

torch.manual_seed(0)
T, C, d = 50, 6, 32                       # 50 time steps, 6 sensor channels, model width d
x = torch.randn(T, C)                     # one window: time by channel

Wq, Wk, Wv = (torch.randn(C, d) for _ in range(3))
Q, K, V = x @ Wq, x @ Wk, x @ Wv          # project each time step to query/key/value

scores = (Q @ K.transpose(0, 1)) / d**0.5 # T x T compatibility between every pair of steps
weights = F.softmax(scores, dim=-1)       # each row is a distribution over source steps
context = weights @ V                      # T x d : each step rebuilt from relevant steps

print("attention-map shape:", tuple(weights.shape))          # (50, 50): time attends to time
print("step 25 looks most at step:", int(weights[25].argmax()))

As Listing 15.1.1 shows, the attention map is a first-class, inspectable object: it tells you, per output moment, which input moments drove the representation, which is the raw material for the interpretability methods of Chapter 67. Treat those maps as evidence, not proof; a confident weight is a correlation the model found useful, not a guaranteed cause.

Attention over channels: which sensors explain each other

Now transpose the question. Treat each sensor channel as a token, with its value across the window (or a learned embedding of it) as the vector to be projected, and let attention run over the channel axis. The weights now form a sensor-by-sensor map: for a nine-axis inertial unit plus a barometer and a magnetometer, the model learns which channels inform which, so the three accelerometer axes can pool into a motion estimate while the magnetometer is down-weighted during an electrical disturbance. This is a learned, input-dependent alternative to the hand-built cross-channel features of Chapter 8, and its great practical virtue is that it does not care about channel order or, within limits, channel count: a model that attends over channels can accept a sensor dropping out by simply masking its key, rather than crashing on a missing input. Whether a given architecture should mix channels at all, or keep them strictly separate, is a genuine design fork with real accuracy and robustness consequences; that decision is the whole subject of Section 15.5, so here we only establish that channel attention is the tool that makes mixing learnable.

import torch, torch.nn as nn

mha = nn.MultiheadAttention(embed_dim=64, num_heads=8, batch_first=True)
x = torch.randn(16, 50, 64)                       # batch, time steps, model width
out, attn = mha(x, x, x, need_weights=True)       # self-attention over the time axis
# feed x.transpose(1, 2) instead to attend over the channel axis

Putting the two axes together

Real sensor transformers rarely attend over one axis alone. The dominant patterns are three. Flatten time and channel into one long sequence of tokens and let full attention couple everything, which is the most expressive and the most expensive, since the score matrix grows with the product of the two axis lengths. Factorize the two axes into alternating blocks, a temporal-attention layer followed by a channel-attention layer, so cost stays linear in each axis while information still crosses both over depth; this axis-factorized design is the workhorse for multichannel sensing and reappears in the spatial-temporal graph models of Chapter 54. Restrict to one axis on purpose, the channel-independent choice, when the channels are better kept separate. The right pattern depends on how strongly your sensors are physically coupled and on your compute budget, and the trilemma of accuracy, latency, and memory it creates is what the rest of this chapter equips you to navigate.

What's Next

In Section 15.2, we confront the quadratic cost head-on by changing what a token is: instead of one token per sample, we chop the stream into short patches and embed each as a single token, cutting the sequence length by the patch size and giving each token a richer local view before attention ever runs.