"Shuffle my tokens and I compute the same answer. That is elegant, until you remember the accelerometer spike came before the airbag fired, not after."
A Permutation-Invariant AI Agent
Why this section matters
Attention, as built in Section 15.1, is a set operation. Feed it the same tokens in a different order and every output is identical, because the softmax over key-query scores has no idea which token came first. That is fatal for sensor data, where order is the signal: a heartbeat is defined by the spacing of its peaks, a fault by a transient that arrives before a shutdown, a gesture by a trajectory through time. It is doubly fatal once you patchify (Section 15.2) a multi-channel stream, because now every token carries two coordinates the model must recover: when it happened and which sensor it came from. Positional encoding is how you hand those coordinates back to a permutation-invariant network. Get it wrong and the transformer treats a 100 Hz IMU burst as an unordered bag of numbers; get it right and it reads time like a clock and channels like a wiring diagram. Sensor streams add a twist the language world rarely faces: samples are often irregularly spaced in real time, so the "position" you must encode is a physical timestamp, not an integer index.
This section assumes the attention mechanism and tokenization of the two preceding sections, the sampling-and-timestamp vocabulary of Chapter 3 (regular versus event-driven sampling, clock skew, jitter), and the notion of a per-channel embedding from Chapter 13. Notation stays consistent with the chapter: a patchified window is a sequence of \(N\) tokens, each an embedding in \(\mathbb{R}^{D}\), and token \(i\) carries a real timestamp \(t_i\) and a sensor (channel) identity \(c_i\).
The permutation-invariance problem, stated precisely
Let \(Z \in \mathbb{R}^{N \times D}\) be the token matrix and \(P\) any permutation matrix. Self-attention satisfies \(\text{Attn}(PZ) = P\,\text{Attn}(Z)\): it is equivariant to reordering, so the set of outputs is unchanged and only their arrangement follows the input. A recurrent or convolutional model (Chapter 14) never had this problem, because recurrence walks the sequence in order and a causal convolution reads a fixed local window; order is baked into the computation. A transformer throws that structure away in exchange for global, parallel mixing, and must be told position explicitly. The standard remedy is to add (or otherwise inject) a positional encoding \(p_i\) to each token so that \(z_i \leftarrow z_i + p_i\) before the first attention layer. The design question this section answers is what \(p_i\) should be when the index \(i\) is a poor proxy for physical time and when tokens also differ in sensor identity.
Encoding time: absolute, relative, and continuous
The original sinusoidal encoding assigns dimension \(2k\) and \(2k+1\) of \(p_i\) the values \(\sin(i/\omega_k)\) and \(\cos(i/\omega_k)\) with geometrically spaced frequencies \(\omega_k = 10000^{2k/D}\). Its what is a smooth, bounded, multi-scale fingerprint of position; its why is that a linear map of \(p_i\) can express any fixed offset, so the model can learn to attend "twelve steps back" as a rotation in encoding space; its how is a fixed lookup requiring no parameters. A learned alternative simply makes \(p_i\) a trainable vector per index, which fits training data better but does not extrapolate past the longest window seen in training, a real limit for variable-length sensor segments.
Two refinements dominate modern practice. Relative encodings inject a function of \(i-j\) directly into the attention score, so the model reasons about gaps rather than absolute clock positions, which matches the fact that a heartbeat interval means the same thing at minute one and minute sixty. Rotary position embedding (RoPE) implements this by rotating the query and key vectors by an angle proportional to position, so the dot product \(q_i^\top k_j\) depends only on \(i-j\); it is now the default in most sequence transformers because it is relative, parameter-free, and length-extrapolates gracefully.
The sensor-specific complication is irregular sampling. If your tokens arrive at real times \(t_i\) that are not evenly spaced (asynchronous channels, event cameras, duty-cycled radios, dropped packets), then encoding the integer index \(i\) tells the model a lie about elapsed time. The fix is continuous-time encoding: feed the real timestamp into the sinusoidal formula, replacing \(i\) with \(t_i\) (in seconds) so that two tokens 40 ms apart get an encoding distance proportional to 40 ms regardless of how many packets fell in between. This is the same idea that lets state-space models and neural ODEs handle irregular streams, and it is the honest way to represent the jittery clocks discussed in Chapter 3.
Encode elapsed time, not token index
For regularly sampled data the index and the timestamp are interchangeable up to a constant, so index-based encoding is fine. The moment sampling becomes irregular, the two diverge, and index-based encoding silently compresses long gaps and stretches short ones. Encoding the physical timestamp \(t_i\) makes the geometry of your attention match the geometry of real time. It also lets a single trained model run at a different sample rate at deployment, because "half a second ago" is expressed in seconds, not in a count of tokens that changes with the rate.
Encoding sensor identity
Once you patchify a multi-channel stream, a token no longer means "the whole system at time \(t\)"; it means "sensor \(c\), around time \(t\)." The model must know \(c\) to interpret the numbers, because a value of 9.8 means gravity on an accelerometer axis and something entirely different on a magnetometer or a skin-temperature probe. The clean solution mirrors time encoding: maintain a learned identity embedding table with one vector per channel and add it to every token from that channel. This is exactly the "variate token" idea that channel-as-token designs exploit, and it is what lets one transformer ingest heterogeneous sensor suites whose channels have no natural ordering. Because identity is categorical (there is no sense in which the gyroscope is "between" the barometer and the thermometer), you use a learned lookup, not a sinusoid. Identity embeddings also give you graceful degradation: if a sensor drops out, you simply omit its tokens, and the surviving tokens still carry correct identity, a property we lean on heavily for missing-modality robustness in Chapter 50. The design choice of whether channels even see each other in attention is the subject of Section 15.5; here we only ensure that when they do, each token knows who it is.
A driver-monitoring unit fusing IMU, CAN speed, and a cabin thermopile
An automotive team builds an in-cabin event classifier from three asynchronous streams: a 200 Hz six-axis IMU, vehicle speed decoded from the CAN bus at roughly 50 Hz, and a slow thermopile array at 8 Hz. They patchify each channel independently and pool all patches into one token sequence. Two encodings are added to every token: a continuous-time sinusoid computed from the token's real timestamp (aligned to a common vehicle clock), and a learned 6-way identity embedding tagging which stream it came from. When a hard-braking event fires, the model must line up the IMU deceleration transient with the CAN speed drop that follows tens of milliseconds later; index-based encoding would have scrambled that lag because the three streams emit wildly different token counts per second. With timestamp encoding the lag is represented in milliseconds, and the identity embedding keeps the slow thermopile from being mistaken for a low-frequency accelerometer axis. The same model later runs unchanged when a firmware update raises the thermopile to 16 Hz, because nothing in the encoding was tied to a fixed rate.
The snippet below builds both encodings for a mixed-rate batch: a continuous-time sinusoid from real timestamps and a learned identity table, then adds them to the token embeddings. It is the arithmetic of this section made executable, and it is deliberately rate-agnostic.
import torch, torch.nn as nn
class TimeAndIdentityEncoding(nn.Module):
def __init__(self, d_model, n_sensors, max_period=10000.0):
super().__init__()
assert d_model % 2 == 0
k = torch.arange(0, d_model, 2).float() / d_model
self.register_buffer("inv_freq", 1.0 / (max_period ** k)) # (d_model/2,)
self.identity = nn.Embedding(n_sensors, d_model) # learned per channel
def forward(self, z, t, c):
# z: (B, N, D) tokens; t: (B, N) real timestamps in seconds; c: (B, N) sensor ids
ang = t.unsqueeze(-1) * self.inv_freq # (B, N, D/2)
time_pe = torch.cat([ang.sin(), ang.cos()], dim=-1) # (B, N, D)
return z + time_pe + self.identity(c)
enc = TimeAndIdentityEncoding(d_model=64, n_sensors=6)
z = torch.randn(2, 10, 64)
t = torch.rand(2, 10).cumsum(-1) # irregular, strictly increasing seconds
c = torch.randint(0, 6, (2, 10)) # which sensor each token came from
out = enc(z, t, c)
print(out.shape) # torch.Size([2, 10, 64])
t, not indices) and a learned per-sensor identity embedding to each token. Swapping t for torch.arange would recover the classic index-based encoding.Putting it together: factorized encodings for the patch grid
After patchification a batch is naturally a grid indexed by (channel, time-patch), much like an image is indexed by (row, column). The proven recipe is a factorized encoding: build a time encoding that depends only on \(t_i\) and an identity encoding that depends only on \(c_i\), and sum them. This keeps the parameter count linear in (sensors + patches) rather than multiplicative, lets the two axes generalize independently (a new sample rate reuses the identity table; a new sensor reuses the time encoding), and mirrors how 2D vision transformers factor row and column encodings. You add both to the token before layer one; from there attention is free to build any cross-term it needs. Resist the temptation to concatenate rather than add unless you have spare width, because addition preserves the model dimension the rest of the stack expects.
RoPE and encoding tables in a few lines
You rarely hand-roll rotary embeddings in production. torchtune.modules.RotaryPositionalEmbeddings or rotary-embedding-torch apply RoPE to your query and key tensors in one call, replacing roughly 30 to 40 lines of index-juggling sin and cos rotation with two lines and handling the interleave-versus-split convention correctly. For learned identity embeddings, nn.Embedding(n_sensors, d_model) is already the one-line primitive; the library value here is RoPE's fiddly rotation math, not the lookup. When you do need continuous timestamps, keep the sinusoid from the snippet above, because most RoPE utilities assume integer positions and will not accept real seconds without adaptation.
Where the field is moving
Time-series-specific position encoding is an active research target. Rotary position embedding (RoPE), introduced for language, is now standard in sensor and time-series transformers because it is relative and extrapolates. Recent work argues that generic encodings underserve time series and proposes dedicated schemes: ConvTran's tAPE (a time-aware absolute encoding whose frequencies are scaled to series length) paired with eRPE (an efficient relative encoding), and irregular-time approaches that feed continuous timestamps directly, as mPLASTIC and time-embedding layers do for asynchronous clinical streams. The through-line is the same lesson as this section: encode physical elapsed time and categorical sensor identity, not raw token index. The state-space models of Chapter 16 sidestep positional encoding entirely by being sequential again, a tradeoff worth weighing.
Exercise
Take a 3-channel IMU window sampled at a nominal 100 Hz but with 5% of samples randomly dropped, and patchify it. (1) Build two versions of the token sequence: one with index-based sinusoidal encoding and one with timestamp-based encoding from the true (gap-aware) times. (2) Construct a synthetic label that depends on the real elapsed gap between two events (for example, "the second peak arrives within 120 ms of the first"). (3) Train a small transformer on each and compare accuracy. Explain, with reference to the key-insight callout, why the timestamp version should win as the drop rate rises.
Self-check
1. Why does adding any fixed \(p_i\) break the permutation equivariance of attention, and why is that the goal rather than a bug?
2. A colleague encodes token index on a stream with heavy packet loss and reports that the model ignores timing. What single change fixes this, and why?
3. Why is sensor identity encoded with a learned lookup table rather than a sinusoid, when time is encoded with a sinusoid?
What's Next
In Section 15.4, we confront the cost that makes all of this hard to scale: self-attention is quadratic in the number of tokens, and sensor windows can be very long. We will look at long-context and efficient-attention techniques (sparse, linear, and windowed attention) that let a transformer keep the positional grounding built here while reading minutes of high-rate signal without the memory bill exploding.