Temporal convolutional networks

"The recurrent net made me read the recording one sample at a time, like a ledger. The convolution let me read the whole page at once and still promise I never peeked at tomorrow."

A Newly Parallelized AI Agent

This section builds directly on the 1D convolution, receptive field, and causal-padding ideas from Chapter 13, and it treats the sample-rate and arrow-of-time constraints as given from Chapter 3. We keep the book's notation: a window is \(x \in \mathbb{R}^{C \times T}\) with \(C\) channels and \(T\) time steps. The detailed arithmetic of how dilation grows the receptive field is the whole subject of the next section, so here we introduce dilation only far enough to explain the architecture and defer the sizing math to Section 14.3.

Three ingredients: causal, dilated, residual

A TCN is defined by combining three properties, each fixing a specific weakness of a naive convolutional stack. Causal convolutions pad only on the left, so the output at time \(t\) depends on inputs at or before \(t\) and never after. This is not decoration: a streaming fall detector or a live arrhythmia monitor must produce \(z[t]\) from the past alone, and a symmetric-padding model that trained on future samples will silently leak the answer (the leakage-safe discipline of Chapter 5 applies to architecture, not just to splits). Dilated convolutions insert gaps of \(d\) samples between the kernel taps, so a length-\(K\) kernel spans \(d(K-1)+1\) input samples while still touching only \(K\) weights. Stacking layers with dilation doubling as \(d = 1, 2, 4, 8, \dots\) makes the receptive field grow exponentially with depth, which is how a compact TCN reaches the seconds-long context a sensor task needs without the linear, parameter-hungry depth a plain stack would demand. Residual connections wrap each pair of dilated convolutions in a skip path, so the block learns a correction rather than a full transform; this keeps gradients healthy through the deep stack that the exponential dilation schedule requires.

The residual block, concretely

The canonical TCN block, from Bai, Kolter, and Koltun's 2018 sequence-modeling study, chains two dilated causal convolutions, each followed by a nonlinearity and a regularizer, then adds the block input back through a skip connection. Written for one block at dilation \(d\),

where \(\mathrm{WN\text{-}Conv}_d\) is a weight-normalized dilated causal convolution and \(\mathrm{Drop}\) is dropout. When the block changes channel count, the skip path uses a \(1\times1\) convolution so the addition is shape-compatible. The why of each piece is worth stating: weight normalization stabilizes the deep stack, dropout on the temporal features guards against a model that memorizes one subject's idiosyncrasy, and the residual add is what lets you keep stacking blocks until the receptive field covers your physics. A full TCN is simply \(B\) of these blocks with dilations \(1, 2, 4, \dots, 2^{B-1}\), optionally repeated for extra capacity, then a task head. That head is a global pool plus a linear layer for sequence-to-label, or a per-step \(1\times1\) convolution for sequence-to-sequence, the distinction we develop in Section 14.4.

The code below builds one residual block and a small TCN, then prints how the receptive field and tensor shape behave. It makes the causal padding explicit by chopping off the extra left-pad samples so the output stays exactly length \(T\), the detail most from-scratch implementations get wrong.

import torch, torch.nn as nn

class Chomp1d(nn.Module):
    def __init__(self, s): super().__init__(); self.s = s
    def forward(self, x): return x[:, :, :-self.s].contiguous() if self.s else x

class TCNBlock(nn.Module):
    def __init__(self, c_in, c_out, k, d, p=0.1):
        super().__init__()
        pad = (k - 1) * d                      # left-pad, then chomp -> causal
        def conv(ci): return nn.Sequential(
            nn.utils.weight_norm(nn.Conv1d(ci, c_out, k, padding=pad, dilation=d)),
            Chomp1d(pad), nn.ReLU(), nn.Dropout(p))
        self.net = nn.Sequential(conv(c_in), conv(c_out))
        self.down = nn.Conv1d(c_in, c_out, 1) if c_in != c_out else None
    def forward(self, x):
        res = x if self.down is None else self.down(x)
        return torch.relu(self.net(x) + res)

class TCN(nn.Module):
    def __init__(self, c_in=1, width=32, blocks=8, k=7, n_classes=4):
        super().__init__()
        layers, ci = [], c_in
        for b in range(blocks):
            layers.append(TCNBlock(ci, width, k, d=2 ** b)); ci = width
        self.tcn = nn.Sequential(*layers)
        self.head = nn.Linear(width, n_classes)
    def forward(self, x):                       # x: (batch, 1, T)
        z = self.tcn(x)                         # -> (batch, width, T), causal
        return self.head(z[:, :, -1])           # last-step readout for a label

x = torch.randn(8, 1, 2048)
rf = 1 + 2 * (7 - 1) * (2 ** 8 - 1)             # kernel 7, 8 doubling dilations
print(TCN()(x).shape, "receptive field:", rf)   # torch.Size([8, 4]) 3061

When to reach for a TCN

A TCN is the strong default when three conditions hold: the discriminative context is bounded and you can name it in milliseconds, you have a GPU and want fast training, and you need the deployment-time predictability of a fixed compute and memory cost per window. It shines on periodic or impulsive sensor signals (gait, vibration, ECG) where the receptive field can be sized to the physics. It is the wrong tool when the relevant context is genuinely unbounded or its length varies wildly, because a TCN can only see as far as it was built to; there an attention model (Chapter 15) or a linear-time state-space model (Chapter 16) will serve better. Whichever you pick, evaluate it with subject-wise, chronological splits: a TCN's causal design earns an honest streaming number only if the split does not hand it future data through the back door.

What's Next

In Section 14.3, we open up the dilation machinery we used here only as a black box: we derive exactly how kernel size, dilation schedule, and depth combine into a receptive field, show how to size that field to a target duration of physics, and expose the coverage gaps a careless dilation schedule leaves inside its own window.