Quantized networks and TinyNAS/MCUNet

"You gave me 256 kilobytes and asked me to see. So I stopped memorizing the whole picture at once and started reading it one tile at a time."

A Resourceful AI Agent

The microcontroller twist: peak SRAM, not model size, is the wall

Chapter 59 taught quantization as four-fold-to-eight-fold compression that also speeds up compute. All of that still holds here, and int8 is the non-negotiable default on a microcontroller: Cortex-M4 and M7 cores have no floating-point throughput to spare, the DSP extensions (Arm Helium, CMSIS-NN) accelerate int8 multiply-accumulate, and eight-bit weights are what fits in flash. What changes on an MCU is which resource runs out first.

A microcontroller has two separate memories with wildly different sizes. Flash (typically 512 kB to 2 MB) is read-only at runtime and holds the program plus the model weights. SRAM (often only 128 kB to 512 kB) is the read-write working memory that must hold the activation tensors while the network computes. Quantizing weights to int8 shrinks the flash footprint. It barely touches the number that actually decides whether the model runs: the peak activation memory, the largest amount of SRAM that must be live at any single instant during a forward pass. For an in-place-friendly kernel that reads an input tensor and writes an output tensor, the memory live at a layer is roughly

The trap is that this peak is dominated by the earliest layers. A vision-scale sensor input is spatially large and only slowly downsampled, so a stem that turns a 96×96×3 image into a 48×48×16 feature map is holding tens of kilobytes when the later layers, deep and narrow, need a fraction of that. The listing below computes this per-layer peak for a MobileNet-style stem and makes the imbalance visible.

def act_kb(c, h, w, bytes_per=1):            # int8 activation => 1 byte / element
    return c * h * w * bytes_per / 1024

# (name, out_channels, out_H, out_W) after each downsampling block
layers = [("conv_stem", 16, 48, 48), ("block1", 24, 48, 48),
          ("block2", 32, 24, 24), ("block3", 64, 12, 12), ("head", 160, 6, 6)]

prev = act_kb(3, 96, 96)                     # 96x96 RGB input tensor
peak = 0.0
for name, c, h, w in layers:
    cur  = act_kb(c, h, w)
    live = prev + cur                        # input kept while output is written
    peak = max(peak, live)
    print(f"{name:10s} live={live:6.1f} KB")
    prev = cur
print(f"peak activation SRAM = {peak:.1f} KB")

Run Listing 61.3.1 and the peak lands on the stem, near 64 kB for a single early transition, even though every weight is already int8. That is the whole problem in one number: you can quantize until the weights are tiny and still be defeated by one spatial feature map. Two strategies attack it from different angles.

TinyNAS: make the memory budget a search constraint, not an afterthought

The first strategy is to stop hand-designing a network and hoping it fits, and instead search for one that fits by construction. Neural architecture search (NAS) automates the choice of depth, width, kernel sizes, and expansion ratios. Naive NAS optimizes accuracy over a fixed, human-chosen search space and then checks the winner against the hardware afterward, which for a 256 kB target almost always fails: the whole search space was implicitly designed for millions of parameters. TinyNAS, the search half of MCUNet (Lin et al., NeurIPS 2020), inverts that. It performs a two-stage, memory-first search.

First, it optimizes the search space itself to the exact resource budget. Given the target's SRAM and flash, TinyNAS sweeps input resolutions and width multipliers and keeps only the configurations whose peak memory and model size fit, then picks the search space whose models place the most FLOPs inside that budget, on the principle that, at a fixed memory cost, a design space that can afford more computation contains more accurate models. Second, it runs a once-for-all style super-network search (following OFA, Cai et al., ICLR 2020) inside that pre-filtered space, training one weight-sharing super-net from which any specialized sub-network can be extracted without retraining. Because the space was memory-filtered up front, every candidate the accuracy search considers already fits the chip. Memory is a hard constraint satisfied by construction, not a penalty term you hope the optimizer respects.

This co-design matters because the sensor and the silicon set the resolution jointly. A person detector on a QVGA image and a keyword spotter on a 40-band log-mel spectrogram (the feature front ends of Section 61.2) have utterly different spatial schedules, so their memory-optimal architectures differ. TinyNAS finds each rather than forcing both through one hand-tuned backbone.

MCUNet and patch-based inference: kill the peak, do not just fit under it

TinyNAS finds the most accurate model under a memory ceiling, but the ceiling is still there, and it is set by that first-layer imbalance from Listing 61.3.1. MCUNetV2 (Lin et al., NeurIPS 2021) attacks the imbalance directly with patch-based inference. Instead of computing a full large feature map for the early, memory-heavy stage, it runs those layers over one small spatial patch at a time, keeping only that patch's activations live, and stitches the results before the network reaches its narrower, memory-cheap deep layers. Because early convolutions are local, a patch plus a thin halo of context is enough to compute its output region correctly. The peak activation of the whole network then collapses to the peak of a single patch, often a four-fold to eight-fold reduction, which is exactly the factor that turns an impossible model into a running one.

Patch-based inference costs some redundant computation in the overlapping halos, so MCUNetV2 pairs it with receptive-field redistribution: it shifts receptive field toward the later stages so the patched early stage stays shallow and the recomputation overhead stays small. The combination is what let MCUNet family models run ImageNet-scale classification and Visual Wake Words person detection on microcontrollers with a few hundred kilobytes of SRAM, a regime that was considered out of reach before. The same trick generalizes past vision to any network whose front end is spatially or temporally wide, including long-window inertial and audio models.

What's Next

In Section 61.4, we move from what to run to what runs it: the microcontroller inference runtimes, LiteRT (TensorFlow Lite Micro), microTVM, and ExecuTorch, that turn a quantized, memory-searched graph into a static, malloc-free binary. We will see how MCUNet's companion TinyEngine plans that peak-activation memory ahead of time, and why the runtime, not just the model, decides your final kilobytes and microseconds.