Sequence-to-label and sequence-to-sequence

"I spent a week improving the encoder, then discovered I had been averaging over the one second where the fall actually happened."

A Belatedly Wiser AI Agent

This section assumes you can build the encoders from Section 14.1 and Section 14.2 and that you understand the receptive-field arithmetic of Section 14.3, because the receptive field determines how much context each per-timestep output can actually see. Prerequisites also include cross-entropy and masked losses from the deep learning refresher (Appendix B) and the subject-disjoint, no-peeking-at-the-future evaluation discipline of Chapter 5. The output topology and the evaluation protocol are joined at the hip, as you will see.

Three output topologies, one encoder

Given an encoder that maps an input sequence \(x_{1:T}\) to hidden states \(h_{1:T}\), there are three ways the output can relate to the input. Sequence-to-label emits a single prediction \(y\) for the whole window: is this ten-second accelerometer clip walking or falling, what is the remaining useful life of this bearing. Aligned sequence-to-sequence emits one label per input step, \(y_{1:T}\), where output \(t\) corresponds to input \(t\): is the person walking at this sample, is this ECG sample inside a QRS complex. Unaligned sequence-to-sequence emits an output sequence \(y_{1:U}\) whose length \(U\) differs from \(T\) and whose steps do not line up with input steps: transcribe a stroke of handwriting into a sequence of characters, forecast the next \(U\) samples from the past \(T\). The encoder can be identical in all three cases. Only the head and the loss change.

Sequence-to-label: pooling a window into one answer

To collapse \(h_{1:T}\) into one vector you need a pooling operator, and the choice matters. The classic RNN move is last-state pooling: take \(h_T\) and discard the rest, on the theory that a recurrent state has already summarized the past. This is cheap and it is what most tutorials show, but it is fragile for sensors: it leans hardest on the most recent samples, so a fall that happens in the middle of a window can be washed out by two seconds of lying still afterward. Mean pooling averages \(h_{1:T}\) and treats every timestep as equally informative, which is robust but dilutes short, sharp events. Max pooling takes the per-feature maximum and is well suited to "did this pattern appear anywhere" detection, such as spotting a single arrhythmic beat. Attention pooling learns a weight \(a_t\) for each timestep and forms \(\sum_t a_t h_t\), letting the model concentrate on the informative instants while remaining a single differentiable head; it is usually the best default and, as a bonus, the weights \(a_t\) are a free saliency map over time, which the interpretability tooling of Chapter 67 can read directly.

A subtlety with variable-length windows: your batch is padded to the longest sequence, so pooling must mask the padding. Averaging over padded zeros silently biases the estimate toward whichever class produces short sequences. The code below shows masked mean pooling and attention pooling side by side.

import torch, torch.nn.functional as F

def masked_mean(h, mask):                      # h: (B,T,D)  mask: (B,T) 1=real 0=pad
    m = mask.unsqueeze(-1).float()
    return (h * m).sum(1) / m.sum(1).clamp(min=1)

def attention_pool(h, mask, score):            # score: Linear(D, 1)
    logits = score(h).squeeze(-1)              # (B,T) one relevance logit per step
    logits = logits.masked_fill(mask == 0, float('-inf'))
    a = F.softmax(logits, dim=1).unsqueeze(-1) # (B,T,1) weights sum to 1 over real steps
    return (a * h).sum(1), a.squeeze(-1)       # pooled vector + saliency over time

B, T, D = 4, 50, 16
h = torch.randn(B, T, D)
mask = torch.ones(B, T); mask[0, 30:] = 0      # first clip is only 30 steps long
pooled_mean = masked_mean(h, mask)
pooled_attn, saliency = attention_pool(h, mask, torch.nn.Linear(D, 1))
print(pooled_mean.shape, pooled_attn.shape, saliency.shape)

The pooled vector then feeds a small classifier or regressor. For classification that is a linear layer plus cross-entropy; for a scalar target such as remaining useful life it is a linear layer plus a regression loss. Whatever the pooling, apply the exact same masking at train and inference time, or the model will see a distribution at deployment it never trained on.

Aligned sequence-to-sequence: one label per timestep

When every timestep carries its own label, drop the pooling entirely and apply the head per step: a shared linear-plus-softmax on each \(h_t\), trained with a cross-entropy averaged over all valid (unpadded) timesteps. This is the natural shape for temporal segmentation tasks: labeling each ECG sample as P-wave, QRS, or T-wave; marking each accelerometer sample as one activity or another for the boundary-precise human activity recognition of Chapter 26; flagging each sample as normal or anomalous, the dense cousin of the change detection in Chapter 12.

The one rule that governs quality here is causality of the receptive field. A per-step output at time \(t\) is only as good as the context it can see. A bidirectional LSTM or a non-causal TCN lets output \(t\) peek at future inputs, which sharpens offline segmentation but is illegal for real-time streaming, where output \(t\) must be emitted before input \(t+1\) exists. The receptive-field budget from Section 14.3 tells you exactly how many past samples each aligned output integrates; if a transition takes longer than that budget to become apparent, the head cannot possibly label its onset correctly, and the fix is a deeper dilation stack, not a fancier loss.

Unaligned sequence-to-sequence: when input and output lengths differ

Some tasks produce an output whose length is not the input's and whose steps do not correspond one-to-one: transcribing an inertial pen trajectory into a string of characters, decoding a burst of surface-EMG into a sequence of intended keystrokes for the neuromotor interfaces of Chapter 27, or forecasting the next \(U\) samples of a signal. Two families handle this. Encoder-decoder models compress the input into a context (a final state or a set of encoder states) and then autoregressively generate the output sequence one step at a time, optionally attending back over the encoder; this is the topology that grows into the attention-based sequence models of Chapter 15. Alignment-free losses, chiefly Connectionist Temporal Classification (CTC), let a per-step encoder emit a longer frame-level label stream (with a blank symbol) and marginalize over all alignments that collapse to the target, so you get a sequence output without ever labeling which input frame produced which output token, exactly the label you cannot afford to collect for handwriting or myoelectric typing.

Forecasting deserves a note because it is the sensor world's most common unaligned task and it hides a leakage trap. Predicting \(x_{T+1:T+U}\) from \(x_{1:T}\) is sequence-to-sequence, and it must be trained and evaluated with a strictly causal split: the target window is the future, and any normalization statistic, any feature, any neighbor drawn from after \(T\) leaks the answer. The horizon \(U\) also interacts with uncertainty: a point forecast \(U\) steps out is nearly useless without a calibrated interval, which is precisely the job of Chapter 18.

What's Next

In Section 14.5, we take the aligned per-step head into the hardest regime it faces: streaming and stateful inference, where the model must emit output \(t\) from a running state before sample \(t+1\) arrives, carry that state across window boundaries without discontinuities, and match its offline-trained behavior exactly on an endless live signal.