Pruning and sparsity

"They gave me ten million weights and told me to recognize a footstep. Nine million of them were just there to watch."

A Recently Downsized AI Agent

What pruning removes, and why the network survives it

Pruning sets a chosen subset of a model's parameters to zero and (ideally) never computes with them again. The premise is over-parameterization: modern networks are trained with far more weights than the final function requires, because the surplus makes optimization easier, not because the deployed model needs it. After training, much of that capacity is dead weight. The classic evidence is that you can rank weights by magnitude \(|w|\), delete the smallest fraction, and lose almost no accuracy, because a weight near zero already contributes almost nothing to any activation. The objective is to find a binary mask \(m \in \{0,1\}^{d}\) that keeps a target fraction of weights while minimizing the loss:

where \(\odot\) is elementwise multiplication and \(k\) is the number of weights you are willing to keep. Solving this exactly is combinatorial, so every practical method is a heuristic for choosing the mask.

The why it works is that a sensor model rarely needs its full expressive budget. A human-activity classifier separating walking from cycling from sitting (Chapter 26) is solving a far lower-dimensional problem than the network it was trained in can represent. Prune the excess and you keep the decision boundary while discarding the scaffolding. The reason the network survives is that the remaining weights can be re-tuned: after masking, a short fine-tuning pass lets the survivors absorb the small errors the deletions introduced. Pruning without that recovery step is what turns a harmless trim into a wound.

Unstructured, structured, and the N:M compromise

Pruning methods differ mainly in the shape of the zeros they create, and that shape decides whether any speedup is real.

Unstructured pruning removes individual weights wherever they are smallest. It reaches the highest sparsity at a given accuracy, because it is unconstrained: it can delete exactly the least useful weight anywhere in the matrix. Its problem is that the surviving weights sit at scattered, irregular positions. A general-purpose CPU or GPU still loads the whole matrix and multiplies through the zeros, so you get a smaller file (great for flash-limited microcontrollers, per Chapter 61) but usually no faster inference unless a specialized sparse kernel is available.

Structured pruning removes whole units: entire channels, filters, attention heads, or rows and columns of a weight matrix. Because it deletes contiguous blocks, the result is simply a smaller dense network that every runtime already accelerates, no special kernels needed. The price is lower peak sparsity: forcing deletions to fall on structural boundaries throws away some useful weights alongside the useless ones. For most edge sensor deployments, structured pruning is the pragmatic default precisely because its speedup is guaranteed on ordinary hardware.

N:M semi-structured sparsity is the modern compromise: within every contiguous block of \(M\) weights, at most \(N\) are nonzero. The 2:4 pattern (two nonzeros in every four) is directly accelerated by NVIDIA Ampere-and-later sparse tensor cores and by a growing set of edge NPUs, delivering close to a \(2\times\) matmul speedup while staying near the accuracy of unstructured pruning. It is regular enough for hardware to exploit, yet fine-grained enough to keep the weights that matter. When your target silicon supports it, N:M is often the best size-accuracy-speed point available.

How to prune: schedules, criteria, and recovery

The naive recipe, "train, delete the smallest weights once, ship," leaves accuracy on the table. Three refinements do the real work.

Gradual, iterative pruning beats one-shot pruning at high sparsity. Instead of removing everything at once, you raise the sparsity target on a schedule across training, interleaving small deletions with fine-tuning so the network continually re-adapts. A widely used schedule ramps sparsity \(s_t\) from an initial \(s_i\) to a final \(s_f\) as

which prunes aggressively early and tapers off, giving the survivors many steps to compensate. The pruning criterion is how you score which weights to cut. Magnitude (\(|w|\)) is the cheap, strong baseline. Better criteria weight each parameter by its estimated effect on the loss, for example \(|w \cdot \partial \mathcal{L}/\partial w|\) (a first-order saliency), which catches small-magnitude weights that nonetheless move the output a lot. Recovery fine-tuning is non-negotiable at anything above modest sparsity: after each pruning step, continue training the unmasked weights so they redistribute the load. Skip it and a 70%-sparse model can collapse; include it and the same model often matches the dense baseline within noise. And because you are re-training and re-evaluating, everything Chapter 65 says about leakage-safe splits applies: measure the recovered accuracy on data the fine-tuning never touched.

import torch, torch.nn.utils.prune as prune

# A tiny sensor classifier; prune its linear layers to 60% sparsity by magnitude.
layers = [(model.fc1, "weight"), (model.fc2, "weight")]

for epoch in range(n_epochs):
    train_one_epoch(model, loader, optimizer)      # normal fine-tuning
    target = 0.6 * min(1.0, (epoch + 1) / warmup)  # ramp sparsity up gradually
    for module, name in layers:
        prune.l1_unstructured(module, name=name, amount=target)  # smallest |w| -> 0
    evaluate(model, val_loader_by_subject)         # leakage-safe split

for module, name in layers:
    prune.remove(module, name)                     # bake the mask into the weights

Listing 59.5.1 shows the pattern in about a dozen lines: ramp the target, re-mask, keep training. Note the evaluation uses a subject-wise validation loader, so the accuracy you recover is not inflated by windows leaking between train and test.

import torch_pruning as tp
example = torch.randn(1, 6, 128)                     # 6-axis IMU window
pruner = tp.pruner.MagnitudePruner(model, example, pruning_ratio=0.4)
pruner.step()                                        # removes filters + repairs all dependents

When pruning pays off, and when it does not

Reach for pruning when the model is over-parameterized for a well-scoped sensing task, when flash or energy is the binding constraint, and when you can afford a fine-tuning pass with a clean evaluation split. It composes cleanly with quantization from Section 59.4: prune first to remove whole units, then quantize the survivors, and the two savings multiply rather than fight. On a wearable running an always-on gesture model, a structured-pruned-then-int8 model can be a quarter of the original size at nearly the original accuracy.

Be skeptical when the claimed win is unstructured sparsity on hardware with no sparse kernel, because there the "90% sparse" headline yields a smaller download and identical latency. Be skeptical too when a model is already tight for its task: pruning a lean TinyML network past a point degrades it fast, since there is no surplus to remove. And always re-validate on a leakage-safe split, because pruning followed by fine-tuning is a fresh training process with fresh opportunities to overfit the very samples you meant to hold out.

What's Next

In Section 59.6, we stop hand-tuning which weights and channels to keep and let the search itself become the model designer: hardware-aware neural architecture search, which optimizes accuracy jointly with the latency, memory, and energy budget of a specific edge target, turning the manual pruning-and-quantizing loop of this chapter into an automated objective.