Quantization and compression

"Sampling decides when I get to look. Quantization decides how many answers the sensor is allowed to give. Compression decides how many of those answers survive the trip to me. I inherit whatever is left."

A Pragmatic AI Agent

This section assumes the sampling picture from Section 3.1 and the noise vocabulary from Chapter 2. It also leans on the notion of information content that Chapter 4 makes precise: entropy is exactly the yardstick that tells you how few bits a stream can be squeezed into. We build the amplitude story first, then the storage story, because the second only makes sense once you understand what the first threw away.

Amplitude quantization: a continuum forced onto a ladder

What it is. An \(N\)-bit converter with full-scale range \(V_{\text{FS}}\) partitions that range into \(2^N\) equal steps. Each step has width \(\Delta = V_{\text{FS}} / 2^N\), the value of the least significant bit (LSB). A true voltage \(v\) is reported as the nearest rung of this ladder, so the reading carries an error \(e = \hat v - v\) bounded by \(\pm \Delta/2\). Twelve bits over a \(\pm 2g\) accelerometer range give \(\Delta \approx 4g / 4096 \approx 0.001g\) per step: that is the finest distinction the sensor can express, no matter how quiet its analog front end.

Why it behaves like noise. For a signal that is busy relative to \(\Delta\) (it crosses many levels between samples), the rounding error \(e\) is well modeled as uniform on \([-\Delta/2, \Delta/2]\), independent from sample to sample. A uniform distribution of width \(\Delta\) has variance \(\Delta^2/12\), so quantization injects an effective noise power of \(\sigma_q^2 = \Delta^2/12\). This is the "quantization noise" that sets a hard floor: you cannot filter below it, because it was created at the moment of rounding and carries no structure to exploit.

How much resolution you actually buy. Comparing full-scale signal power to that noise floor gives the classic signal-to-quantization-noise ratio for a full-scale sinusoid:

Every added bit buys about 6 dB of headroom, roughly one extra factor of two in amplitude resolution. That single formula is the entire economics of bit depth: it tells you a 16-bit ECG channel tops out near 98 dB of dynamic range on paper, and it warns you that if your analog noise already sits 40 dB below full scale, the bottom eight bits are just digitizing that noise. The honest resolution of a real channel is captured by its effective number of bits (ENOB), which folds analog noise and distortion back in and is almost always smaller than the nameplate \(N\).

Dithering: the counterintuitive fix. When a signal is small relative to \(\Delta\), quantization stops looking like friendly noise and starts looking like distortion: a slow ramp gets rendered as flat plateaus with sudden stair-steps, and those steps are harmonically related to the signal, so they survive averaging and masquerade as real structure. Adding a tiny amount of noise (dither) of order \(\Delta\) before quantizing breaks the correlation, trading a visible staircase for a benign broadband hiss you can then average down. It feels wrong to add noise to reduce error, yet it is standard practice in audio and instrumentation converters for exactly this reason.

Non-uniform quantization: spend bits where the signal lives

What it is. A uniform ladder wastes resolution when the signal's amplitude distribution is heavy-tailed: most samples are small, yet a uniform quantizer reserves as many levels for the rare large excursions as for the common small ones. Non-uniform quantization allocates fine steps where samples are dense and coarse steps where they are sparse. The workhorse trick is companding: pass the signal through a compressive nonlinearity (logarithmic \(\mu\)-law or A-law in telephony), quantize the result uniformly, and expand on the far side. An 8-bit \(\mu\)-law channel delivers the perceptual dynamic range of roughly 12 uniform bits for speech, because human loudness perception is itself logarithmic.

Why it matters for sensors. Many physical measurands share this heavy-tailed shape. Vibration is quiet most of the time and spikes at faults; audio and PPG have long calm stretches punctuated by events. Allocating quantization levels on a log or perceptual scale keeps small-signal fidelity where the interesting structure hides while still surviving the rare large event without clipping. The cost is that the quantization noise is now signal-dependent, which any calibration or uncertainty model downstream must account for rather than assume a flat \(\Delta^2/12\).

Compression: the same information in fewer bits

Quantization decided how many distinct values exist; compression decides how compactly the resulting stream of values is stored. Two families matter, and confusing them is a classic sensor-pipeline bug.

Lossless compression reconstructs the exact integers you fed in. It exploits redundancy: successive sensor samples are highly correlated, so instead of storing each absolute value you store the small difference from a prediction (delta or linear-predictive coding), then hand those small residuals to an entropy coder (Huffman, arithmetic, or range coding) that spends few bits on the common near-zero residuals and more on the rare large ones. The entropy of the residual stream from Chapter 4 is the theoretical floor; FLAC does this for audio, and the same predict-then-entropy-code recipe compresses slowly varying industrial telemetry by three to five times with zero fidelity loss. Because it is exact, lossless compression never changes what a model sees; it only changes the bandwidth bill.

Lossy compression throws away information deemed unimportant to buy far larger savings: quantizing transform coefficients (as MP3 and JPEG do), dropping high-frequency detail, or downsampling. The ratios are dramatic (10x and beyond), but the discarded information is gone, and its absence is structured, not random. This is where sensor AI gets bitten.

The short simulation below makes the amplitude story concrete: it quantizes a sinusoid at several bit depths, measures the achieved SQNR, and checks it against the \(6.02N + 1.76\) prediction.

import numpy as np

def quantize(x, n_bits, v_fs):
    step = v_fs / (2 ** n_bits)                 # LSB width
    q = np.round(x / step) * step               # snap to nearest rung
    return np.clip(q, -v_fs / 2, v_fs / 2)      # respect the rails

t = np.linspace(0, 1, 20000, endpoint=False)
x = 0.5 * np.sin(2 * np.pi * 7 * t)             # full-scale sinusoid, V_FS = 1.0

for n in (4, 8, 12, 16):
    xq = quantize(x, n, v_fs=1.0)
    noise = xq - x
    sqnr = 10 * np.log10(np.mean(x ** 2) / np.mean(noise ** 2))
    print(f"{n:2d} bits: measured SQNR {sqnr:5.1f} dB   "
          f"predicted {6.02 * n + 1.76:5.1f} dB")

import numcodecs, numpy as np

samples = (x * 32767).astype(np.int16)          # 16-bit ECG-like stream
codec = numcodecs.Delta(dtype='i2')             # predict: store differences
blosc = numcodecs.Blosc(cname='zstd', clevel=5) # entropy-code the residuals
packed = blosc.encode(codec.encode(samples))
back = codec.decode(blosc.decode(packed))
assert np.array_equal(samples, back)            # exactly lossless
print(f"ratio: {samples.nbytes / len(packed):.1f}x")

What's Next

In Section 3.6, we assemble quantized, compressed sample streams into the units a model actually consumes: buffers, windows, and frames. Quantization and compression decided how faithful each sample is and how cheap it is to move; framing decides how those samples are grouped in time before they reach a feature extractor or a network, closing the loop from raw acquisition to model-ready tensors. The bit-budget decisions here also foreshadow the on-device tradeoffs of Chapter 59, where quantizing the model rather than the signal follows the very same 6-dB-per-bit logic.