"I remember everything, which is exactly the problem. The trick was learning what to forget on purpose."
A Gated AI Agent
Prerequisites
This section assumes you are comfortable with sampled signals and the notion of a fixed-rate stream from Chapter 3, and with the neural-network mechanics (backpropagation, activations, minibatch training) refreshed in Appendix B and applied to sensor windows in Chapter 13. It is useful to have met the state-carrying, one-step-at-a-time structure of a recursive filter in Chapter 9, because a recurrent network is that same idea with a learned, nonlinear update rule. No prior exposure to sequence models is required.
The Big Picture
A recurrent neural network processes a sensor stream the way a Kalman filter does: it keeps a compact summary of everything seen so far, a hidden state, and updates it one sample at a time. Unlike the Kalman filter, it does not know the physics; it learns the update rule from data. This is a natural fit for sensing, where signals arrive as an ongoing stream and the meaning of the current sample depends on what came before it. The plain recurrent cell captures this idea in three lines of algebra but is notoriously hard to train over long horizons because gradients vanish. The LSTM and GRU are the two engineering fixes that made recurrence practical, each adding gates that let the network decide what to remember, what to overwrite, and what to expose. This section builds all three from the recurrence up, shows exactly why the plain version forgets, and gives you the judgment to pick the right cell for a sensor task.
The recurrent idea: a state that carries the past forward
A feedforward window model looks at a fixed slice of the stream and treats every slice independently. That works, but it forces you to commit to a window length up front and it throws away the connection between one window and the next. A recurrent network takes the opposite stance. It reads the stream sample by sample and maintains a hidden state vector \(h_t\) that is meant to summarize the relevant history. At each step it combines the new input \(x_t\) with the previous state to produce the next state:
$$h_t = \tanh(W_x x_t + W_h h_{t-1} + b)$$
The same weight matrices \(W_x\) and \(W_h\) are reused at every time step, which is the temporal analogue of the weight sharing a convolution performs across space. That sharing is what lets one small network handle sequences of any length and is the reason an RNN has a fixed parameter count no matter how long the stream runs. The why is compression: \(h_t\) is a learned, lossy encoding of the past, and the network is trained so that whatever it chooses to keep is exactly what the task needs. For a wearable classifying gait, \(h_t\) might come to encode the phase of the current stride; for a fault detector it might encode how long a vibration has been elevated. The recurrence is trained by backpropagation through time: you unroll the loop across the sequence and treat it as a very deep feedforward network that happens to share weights across its layers.
Key Insight
A recurrent network is a state-space model whose transition function is learned rather than derived. The Kalman filter of Chapter 9 writes down \(h_t = A h_{t-1} + B x_t\) from known dynamics and gives you calibrated uncertainty for free. An RNN replaces \(A\) and \(B\) with a nonlinear learned map, buying the ability to model dynamics you cannot write down, at the cost of the interpretability and the built-in uncertainty. Choosing between them is the classical-versus-learned tradeoff that runs through this whole book, viewed one sample at a time.
Why plain RNNs forget: vanishing and exploding gradients
The elegance of the shared recurrence hides a defect that kept RNNs marginal for years. When you backpropagate the loss at step \(T\) back to an input at step \(t\), the gradient is a product of many Jacobians, one per step in between, each carrying a factor that involves \(W_h\) and the derivative of \(\tanh\). Multiplying \(T - t\) such factors together makes the gradient behave like a number raised to a large power: if the repeated factor has magnitude below one, the gradient shrinks toward zero (it vanishes); if above one, it grows without bound (it explodes). The practical consequence is that a plain RNN cannot learn dependencies that span more than a few dozen steps, because the error signal linking a late output to an early cause has decayed to nothing before it arrives.
For sensing this is a real ceiling, not a corner case. A five-second event sampled at 100 Hz is 500 time steps; a slowly developing bearing fault or a sleep-stage transition spans thousands. The exploding side is easier to patch (you clip the gradient norm to a threshold, which the training recipes of Section 14.7 treat in detail), but vanishing is structural: no learning-rate trick recovers a signal that has already decayed to numerical noise. This is precisely the problem the gated cells were invented to solve.
The listing below makes the decay concrete. It feeds a long random sequence through a plain RNN cell and an LSTM cell, then measures how much the loss at the final step still depends on the very first input.
import torch, torch.nn as nn
torch.manual_seed(0)
T, dim = 200, 16 # a 200-step sequence, e.g. 2 s of 100 Hz sensor data
x = torch.randn(1, T, dim, requires_grad=True)
for name, cell in [("vanilla RNN", nn.RNN(dim, dim, batch_first=True)),
("LSTM", nn.LSTM(dim, dim, batch_first=True))]:
x.grad = None
out, _ = cell(x) # run the recurrence over all T steps
out[:, -1, :].sum().backward() # loss depends only on the LAST step's output
# how strongly does the final-step loss still 'feel' the very first input?
first_step_grad = x.grad[:, 0, :].abs().mean().item()
print(f"{name:12s} gradient reaching step 0: {first_step_grad:.2e}")As Listing 14.1.1 shows, the difference is not a matter of tuning. The plain recurrence attenuates the long-range gradient by construction, and the LSTM's architecture is what preserves it. The next two subsections explain how.
LSTM: gates and a protected cell state
The Long Short-Term Memory cell adds a second state vector, the cell state \(c_t\), that acts as a memory conveyor belt. Information flows along \(c_t\) with only gentle, mostly additive edits, so gradients travel back along it without the repeated multiplicative shrinkage that kills the plain RNN. Three learned gates, each a sigmoid producing values between zero and one, control the edits. The forget gate \(f_t\) decides how much of the old memory to keep, the input gate \(i_t\) decides how much new candidate information \(\tilde{c}_t\) to write, and the output gate \(o_t\) decides how much of the memory to expose as the visible hidden state:
$$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t, \qquad h_t = o_t \odot \tanh(c_t)$$
where \(\odot\) is elementwise multiplication and each gate is computed from \(x_t\) and \(h_{t-1}\). The why of the design is entirely in that first equation: when the forget gate is near one and the input gate near zero, the cell copies its memory forward unchanged, giving a near-perfect gradient path (the "constant error carousel" in the original formulation). The network learns, per dimension and per time step, when to hold a value steady across hundreds of steps and when to let it change. For a clinical monitor tracking whether a patient has been tachycardic, one cell dimension can latch that fact and carry it for minutes while other dimensions track the fast beat-to-beat signal.
In Practice: an LSTM watching a haul-truck engine
A mining fleet operator instrumented its haul trucks with vibration, oil-temperature, and load sensors and wanted early warning of turbocharger degradation. A window CNN flagged failures only once they were loud, which was too late to schedule maintenance. The failure signature was a slow drift: elevated vibration that mattered only if it persisted and coincided with high load over many minutes, a dependency far longer than any practical convolution window. An LSTM fixed the problem because its cell state could accumulate the persistence evidence: one memory dimension effectively counted how long vibration had stayed above baseline under load, holding that count steady across the quiet stretches between rough patches of road. The forget gate learned to reset the count when the truck idled. This is the prognostics pattern developed fully in Chapter 36, and it is the kind of long-horizon, stateful memory that gating exists to provide. As always, the model was evaluated with truck-disjoint splits so a leak of one vehicle's data into both train and test could not inflate the lead time.
GRU: fewer gates, comparable power
The Gated Recurrent Unit is a streamlined cell that keeps the gating idea but drops the separate cell state. It merges the forget and input gates into a single update gate \(z_t\) that interpolates between the old state and a new candidate, and adds a reset gate \(r_t\) that controls how much past state feeds the candidate:
$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$
The convex combination is the crucial line: when \(z_t\) is near zero the state is copied forward unchanged, giving the GRU the same protected gradient path as the LSTM through a leaner mechanism. A GRU has three weight matrices where an LSTM has four, so it uses roughly 25 percent fewer parameters and runs a little faster for the same hidden size. On many sensor benchmarks the two are within noise of each other; the GRU tends to win when data or compute is tight, and the LSTM's extra capacity can pull ahead on the longest, most complex dependencies. For an edge deployment on a microcontroller (Chapter 61), the GRU's smaller footprint is often the deciding factor.
The Right Tool
Writing an LSTM by hand means implementing four gate projections, the cell-state update, the hidden-state readout, and correct minibatch handling of variable-length sequences, roughly forty lines that are easy to get subtly wrong (a swapped gate or a missing bias initialization can silently cripple training). Every framework ships all three cells as one line, fused and GPU-accelerated:
import torch.nn as nn
rnn = nn.RNN(input_size=6, hidden_size=64, num_layers=2, batch_first=True)
lstm = nn.LSTM(input_size=6, hidden_size=64, num_layers=2, batch_first=True)
gru = nn.GRU(input_size=6, hidden_size=64, num_layers=2, batch_first=True)Choosing a cell for a sensor stream
The decision is rarely "RNN, LSTM, or GRU" in isolation; it is a set of coupled choices. Reach for a plain RNN essentially never for real tasks, except as a teaching baseline or when the dependency is provably short and latency is critical. Between the gated cells, start with a GRU as the default (fewer parameters, faster, usually as accurate) and switch to an LSTM only if a validation comparison shows it winning on your longest dependencies. Set the hidden size by the complexity of the state you expect to track, not by reflex; sensor tasks often do well with 32 to 128 units, far smaller than language models. Stack two layers before you consider three, and make the network bidirectional only for offline analysis, because a bidirectional cell reads the whole sequence including the future and therefore cannot run in a live stream, a constraint that Section 14.5 makes central to streaming deployment. Finally, remember that recurrence is now one option among several: the temporal convolutions of Section 14.2, the transformers of Chapter 15, and the linear-time state-space models of Chapter 16 all compete for the same jobs, and the last of these is best read as the gated RNN's idea rebuilt for parallel hardware.
Exercise
Build a synthetic "delayed recall" task: a sequence of random vectors where the label is a copy of the very first vector's sign, so the network must carry information from step 0 to step \(T\). (1) Train a plain nn.RNN, an nn.GRU, and an nn.LSTM at \(T = 20, 100, 500\) and plot final accuracy against \(T\). (2) At which \(T\) does the plain RNN collapse to chance while the gated cells still solve the task? (3) Add gradient-norm clipping to the RNN and report whether it helps the vanishing problem, the exploding problem, both, or neither, and explain why in terms of the Jacobian-product argument.
Self-Check
1. State the update equations for the plain RNN state and the LSTM cell state side by side, and point to the exact term in the LSTM equation that keeps long-range gradients from vanishing.
2. A GRU has fewer parameters than an LSTM of the same hidden size. Which gate combination is responsible for the reduction, and what capability, if any, do you give up?
3. Your task is a live wearable classifier that must emit a label every 20 ms as data arrives. Explain why a bidirectional LSTM is disqualified regardless of its offline accuracy.
What's Next
In Section 14.2, we set recurrence aside and build temporal convolutional networks, which attack the same long-sequence problem from the opposite direction: instead of carrying a state through time, they stack causal convolutions to see far into the past all at once, trading the RNN's sequential memory for parallelism and a fixed, analyzable receptive field.