Practical training recipes

"My loss went to zero, my val curve looked gorgeous, and then a colleague pointed out that all my validation windows came from subjects I had already trained on."

A Chastened AI Agent

This section assumes the encoders and heads of Sections 14.1 through 14.4, the batching and masking of variable-length windows, and the normalization and windowing pipeline of Chapter 13. It leans hard on one non-negotiable discipline: the subject-disjoint, no-peeking-at-the-future evaluation of Chapter 5. Every recipe below is downstream of getting the split right, because a tuning decision measured on a leaky validation set optimizes for a number that will not survive deployment.

The loss is the specification, not a default

The single most consequential training choice for sensor models is the loss, because sensor labels are almost always imbalanced. A fall detector sees minutes of walking for every fraction of a second of falling; a bearing runs healthy for months before it fails; an arrhythmia is a handful of beats in a day-long ECG. Train plain cross-entropy on that and the optimizer discovers the cheapest possible solution: always predict "normal", collect 99.7% accuracy, and never fire. The fix is to make the rare class expensive to miss. Class-weighted cross-entropy multiplies each class's loss by a weight inversely proportional to its frequency. Focal loss goes further, down-weighting easy, confidently-correct samples by a factor \((1-p_t)^\gamma\) so the gradient concentrates on the hard minority, which for detection tasks usually beats static class weights. For heavily skewed segmentation, soft Dice or Tversky losses optimize overlap directly and are far less seduced by the majority background than per-pixel cross-entropy.

Optimization for sequences: clip, warm up, decay

Recurrent networks propagate gradients through as many multiplications as there are timesteps, so a window of \(T=500\) samples is a 500-deep computation graph. That depth makes the gradient norm volatile: one bad batch can produce a gradient a thousand times larger than the running average and blow the weights out in a single step. Gradient clipping is the cheap, load-bearing defense. Clip the global gradient norm to a fixed ceiling (a value of 1.0 to 5.0 is a good starting range) before every optimizer step, and the training that was diverging on batch 40 becomes boring and stable. TCNs are less prone to this than LSTMs because their gradients flow through a fixed convolutional depth rather than an unrolled recurrence, but clipping costs nothing and protects both.

AdamW is the default optimizer for these models: it adapts per-parameter step sizes, which suits the heterogeneous gradient magnitudes of gated recurrent cells, and its decoupled weight decay is a cleaner regularizer than Adam's L2 coupling. Pair it with a schedule that warms up the learning rate over the first few hundred steps, then decays it, typically with a cosine curve. Warmup matters because a large initial learning rate on a freshly initialized recurrent state produces exactly the gradient explosions clipping is fighting; easing in lets the normalization statistics and gate biases settle first. The training loop below wires clipping, warmup-plus-cosine, AdamW, and mixed precision into the few lines that carry most of the stability.

import torch, math
from torch.optim import AdamW

def make_scheduler(opt, warmup, total):
    def lr_lambda(step):
        if step < warmup:                       # linear warmup
            return step / max(1, warmup)
        p = (step - warmup) / max(1, total - warmup)
        return 0.5 * (1 + math.cos(math.pi * p)) # cosine decay to 0
    return torch.optim.lr_scheduler.LambdaLR(opt, lr_lambda)

opt   = AdamW(model.parameters(), lr=3e-3, weight_decay=1e-2)
sched = make_scheduler(opt, warmup=300, total=10_000)
scaler = torch.cuda.amp.GradScaler()            # mixed precision

for x, y, mask in train_loader:
    opt.zero_grad()
    with torch.cuda.amp.autocast():
        loss = masked_loss(model(x), y, mask)   # ignore padded steps
    scaler.scale(loss).backward()
    scaler.unscale_(opt)
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    scaler.step(opt); scaler.update(); sched.step()

Regularization and augmentation that respect physics

Sensor models overfit fast because labeled sensor data is scarce, so regularization earns its keep. Weight decay and dropout are the baseline, but recurrent dropout must be applied with the same mask across timesteps (variational dropout) rather than resampled each step, or it corrupts the recurrence itself. The higher-leverage lever is data augmentation, and here sensor work diverges sharply from vision. An augmentation is only legal if the label survives it. Adding Gaussian noise, small time warps, magnitude scaling, and random cropping of windows all preserve "this is walking". But flipping the sign of an accelerometer axis inverts gravity and turns walking into physical nonsense; time-reversing an ECG destroys the causal P-QRS-T ordering; rotating a three-axis IMU is valid only if you rotate all three channels by the same rotation matrix, because the axes are physically coupled. The measurement models of Chapter 2 tell you which transforms are label-preserving; borrow a vision augmentation blindly and you train the model on data that cannot occur.

Windowing, batching, and honest validation

Two mechanical choices decide whether your batches are even coherent. Window length must exceed the receptive field of Section 14.3 and must contain the temporal pattern the label describes; a two-second window cannot learn a gait cycle that takes three. Batching variable-length sequences requires padding to the longest member and a mask so the loss ignores the pad, exactly the masked loss the code above assumes; bucket sequences of similar length together to keep padding waste low. For very long sequences that will not fit in memory, truncated backpropagation through time splits the sequence into chunks, carries the recurrent state forward across chunk boundaries (detached from the graph), and backpropagates only within each chunk. Set the truncation length to comfortably exceed the longest dependency you need the model to learn, because gradients never flow past the truncation boundary.

The recipe that overwhelmingly determines the deployed number is the split. Random-window splits leak because consecutive windows from one subject or one machine are nearly identical, so a window in validation has a near-twin in training. Split by subject, by device, or by time period, never by window, and normalize using statistics computed on the training split alone, then applied unchanged to validation, so no test-set statistic ever touches training. Early-stop on the leakage-safe validation metric, and treat the resulting curve as the truth. The formal protocols for this live in Chapter 65, and when labels are scarce enough that no split leaves you enough to train on, the self-supervised pretraining of Chapter 17 is the recipe that buys back label efficiency.

What's Next

In Chapter 15, we replace the recurrence and the fixed convolution with attention over time and channels, which lets a model relate any two timesteps directly, at a compute cost that forces its own set of training recipes: patchification, positional encoding for sensor identity, and the efficient-attention tricks that keep long sensor sequences affordable.