Quantization (int8, int4, and hardware-native 2/4-bit)

"They asked me to memorize the world in 32 bits per weight. I countered with 4, kept my accuracy, and pocketed the other 28 as battery life."

A Thrifty AI Agent

What quantization is: mapping reals onto a small integer grid

Quantization is a linear map from a continuous range of real numbers onto a finite, evenly spaced integer grid. The standard affine (asymmetric) scheme picks a positive scale \(s\) and an integer zero-point \(z\), then represents a real value \(r\) by the integer

For signed int8 the grid is \([q_{\min}, q_{\max}] = [-128, 127]\), so 256 levels stand in for the whole dynamic range of a tensor. The scale \(s\) is the width of one grid step; the zero-point \(z\) is the integer that maps exactly to real zero, which matters because padding and ReLU outputs make exact zero extremely common and you do not want to represent it approximately. When a tensor is naturally symmetric about zero (most weights are), you fix \(z = 0\) and use symmetric quantization, which drops the zero-point subtraction from the inner loop and is what almost every accelerator prefers for weights.

Why does this help so much? Two independent wins stack. First, memory and bandwidth: an int8 weight is a quarter the size of an fp32 weight, so the model file, the SRAM footprint, and the traffic across the memory bus that Section 59.3 flagged as the real energy sink all shrink four-fold. Second, compute: integer multiply-accumulate units are smaller, faster, and far cheaper in energy per operation than floating-point ones, which is exactly why the NPUs and DSPs of Section 59.2 expose int8 (and increasingly int4) datapaths as their fast path. A quantized matmul accumulates products of int8 values into int32, then a single rescale returns to the next layer's int8 domain.

Choosing the range: the accuracy lives in the clipping

Everything hinges on choosing \(s\) and \(z\), which is equivalent to choosing the real interval \([r_{\min}, r_{\max}]\) that the grid covers. Set the range too wide and every step is coarse, so small but important values round to the same integer. Set it too narrow and large values saturate at the clip bound, destroying outliers that may be doing real work. This is a classic estimation tradeoff, resolution against clipping, and it is why range selection (often called calibration) is the part of quantization that actually decides your accuracy.

Weights are easy: you know them at export time, so you just read off their min and max (per channel, see below). Activations are the hard part, because their range depends on the input data and you only learn it by running representative samples through the network. Two families of decision matter:

• Granularity. Per-tensor quantization uses one \((s, z)\) for a whole weight tensor; per-channel gives each output channel its own scale. Per-channel costs a vector of scales instead of a scalar and almost always recovers most of the int8 accuracy gap on convolutional and depthwise layers, whose channels have wildly different magnitudes. Use per-channel for weights by default.
• Calibration statistic. Plain min/max is fragile to a single outlier activation. Better estimators clip the range to minimize information loss: a histogram-based search that minimizes the Kullback-Leibler divergence between the float and quantized distributions, or a percentile cutoff (say the 99.99th) that deliberately saturates the rarest extremes to sharpen the grid for the bulk of values.

PTQ and QAT: two ways to pay for the accuracy

There are two routes to a quantized model. Post-training quantization (PTQ) takes a finished float model, runs a few hundred unlabeled calibration batches to fix activation ranges, and emits integers. It needs no gradients, no labels, and minutes of compute, and for int8 it usually just works. When it does not, or when you push to int4 and below, quantization-aware training (QAT) inserts "fake quantization" nodes into the graph during a short fine-tune: the forward pass rounds exactly as the deployed integer kernel will, so the loss sees the rounding error, while the backward pass uses a straight-through estimator to pass gradients through the non-differentiable rounding step. QAT teaches the weights to be robust to their own quantization and routinely recovers the last point or two of accuracy that PTQ leaves on the table, at the cost of a training loop and labels.

The decision rule is simple. Reach for PTQ first; it is nearly free. Escalate to QAT only when PTQ's held-out metric misses your target, which happens most often at int4, on tiny models with little redundancy to spare, or on tasks where a few misclassified samples are expensive. The listing below shows the PTQ path in modern PyTorch, where the whole calibrate-then-convert flow is a handful of lines.

import torch
from torch.ao.quantization.quantize_fx import prepare_fx, convert_fx
from torch.ao.quantization import get_default_qconfig_mapping

model.eval()
qmap = get_default_qconfig_mapping("x86")          # per-channel int8 weights
example = torch.randn(1, 3, 128)                    # one sensor window, shape (B, C, T)
prepared = prepare_fx(model, qmap, example)         # insert observers

with torch.no_grad():                               # calibration: NO labels needed
    for window in calib_loader:                     # ~200 representative windows
        prepared(window)                            # observers record activation ranges

int8_model = convert_fx(prepared)                   # fold ranges -> real int8 kernels
torch.save(int8_model.state_dict(), "model_int8.pt")

As Listing 59.4.1 shows, the entire post-training path is observe, calibrate, convert. Note what is absent: no labels, no loss, no optimizer. That is the whole appeal of PTQ, and why it is always the first thing to try.

Int4 and hardware-native 2/4-bit: when four bits is the datapath, not a trick

Int8 is now the boring default. The frontier that matters for sensor edge AI is sub-byte. Int4 halves the model again versus int8, which is decisive when weights, not activations, dominate the footprint (the usual case once you are on-chip). Naive int4 PTQ often loses too much, so two ideas carry it: weight-only quantization (keep activations at int8 or fp16, quantize only the bulky weights, so the sensitive activation path stays precise), and group-wise scales (a separate scale every 32 or 64 weights along a row, which tames the outlier weights that otherwise force a coarse grid). Weight-only int4 with group size 128, popularized by GPTQ and AWQ for language models, transfers directly to the larger sensor and wearable foundation models of Chapter 20.

The crucial word is hardware-native. A format only buys speed if the silicon has a datapath for it; otherwise the runtime unpacks your 4-bit weights back to int8 or fp16 before every multiply and you save memory but not a single cycle of compute. Recent NPUs and DSPs changed this: many Arm Ethos-U and vendor accelerators expose true 4-bit weight datapaths, and some neuromorphic and TinyML parts go to 2-bit or ternary. When 2/4-bit is native, sub-byte weights cut both bandwidth and multiply energy; when it is emulated, they cut only bandwidth. Always confirm which case you are in against the accelerator's datasheet before promising a latency number, because the two outcomes differ by more than 2x.

What's Next

In Section 59.5, we attack the model from the other direction: instead of shrinking every weight's precision, we remove whole weights and channels through pruning and structured sparsity, and see how sparsity and quantization compose into a single edge-optimization budget.