CPU, GPU, NPU, DSP, MCU tradeoffs

"They gave me five kinds of silicon and one power budget, then acted surprised when I had opinions about which one wakes up first."

A Frugal AI Agent

This section assumes you are comfortable with the arithmetic of a neural network layer as a pile of multiply-accumulate operations, the material from Chapter 13, and with the idea that numeric precision is a design knob, which we develop fully in Section 59.4 on quantization. We will speak in terms of MACs (multiply-accumulate operations), bytes moved through memory, and two hardware numbers every datasheet reports: peak throughput and peak memory bandwidth. We keep the memory, power, and thermal envelope itself for Section 59.3; here the concern is the compute fabric.

Five engines, one spectrum from flexible to frugal

The five processor classes are not five arbitrary products; they sit on a single axis that trades programmability against energy efficiency. Understanding where each sits tells you what it is for.

The CPU is the generalist. It runs arbitrary control flow, branches, and any operator you can compile, including the odd preprocessing step or custom layer no accelerator supports. Modern embedded CPUs (Arm Cortex-A, and the vector-extended Cortex-M) add SIMD (single-instruction-multiple-data) units that process several samples per instruction, so a CPU is rarely the fastest option but is almost always the fallback that guarantees a model runs at all. Its weakness is energy: general-purpose pipelines spend most of their transistors on instruction fetch, decode, and speculation, not on arithmetic.

The GPU trades that flexibility for throughput. Its SIMT (single-instruction-multiple-thread) design runs thousands of lightweight threads over dense, regular tensors, which is why it dominates training and heavy vision workloads. On a phone or a robot the GPU is the engine for large convolutional and transformer blocks (Chapter 15), but it is power-hungry and has meaningful launch latency, so waking it for a tiny always-on classifier is wasteful.

The NPU (also called a TPU, neural engine, or matrix accelerator) is a fixed-function array of MAC units, typically a systolic array, built to do one thing: dense low-precision matrix multiply at extreme efficiency. NPUs report throughput in TOPS (tera-operations per second) and, more tellingly, in TOPS per watt, where they beat a CPU by one to two orders of magnitude. The catch is operator coverage: an NPU runs the operators its compiler supports, in the precisions it supports (usually int8, increasingly int4), and anything outside that set is evicted back to the CPU. A model that looks tiny can run slowly because one unsupported activation forces a round trip off the accelerator every layer.

The DSP is the specialist for streaming signal math: FIR/IIR filters, FFTs, and the fixed-point vector loops of an audio or radar front end. DSPs use VLIW (very-long-instruction-word) and SIMD designs to sustain many MACs per cycle at very low power, and crucially they are efficient on the always-on duty cycle, staying awake to watch a microphone or an IMU while the big cores sleep. Many recent DSPs also run quantized neural nets, blurring the line with NPUs.

The MCU is the floor of the hierarchy: a self-contained chip with kilobytes to a few megabytes of RAM, no operating system required, microamp sleep currents, and a unit cost measured in cents. It is the home of TinyML (Chapter 61 develops this fully). An MCU will not run a large model, but it will run a 30 KB keyword spotter for a year on a coin cell, which no other engine here can claim.

The roofline: is your layer compute-bound or memory-bound?

Peak TOPS is a seductive and often useless number, because most sensor-model layers never reach it. Whether a layer is limited by arithmetic or by moving data is decided by its arithmetic intensity \(I\), the number of operations performed per byte fetched from memory:

$$ I = \frac{\text{operations}}{\text{bytes moved}}, \qquad I^{\star} = \frac{\text{peak throughput (ops/s)}}{\text{memory bandwidth (bytes/s)}}. $$

The ridge point \(I^{\star}\) is a property of the hardware alone. When a layer's \(I < I^{\star}\), the engine finishes its math faster than memory can feed it, so the layer is memory-bound and adding TOPS buys nothing. When \(I > I^{\star}\), the layer is compute-bound and faster memory buys nothing. The achievable rate is the roofline: \(\min(\text{peak throughput},\ I \times \text{bandwidth})\). This single inequality explains why an NPU rated at 10 TOPS can crawl on a depthwise convolution (very low \(I\), so memory-bound) while flying on a dense fully connected layer (high \(I\), so compute-bound). It also explains why quantization helps twice: int8 weights are a quarter the bytes of float32, which both raises \(I\) and lowers the bandwidth wall.

def roofline(layer_macs, bytes_moved, peak_tops, bandwidth_gbps):
    ops = 2 * layer_macs                       # 1 MAC = 1 multiply + 1 add
    intensity = ops / bytes_moved              # operations per byte
    ridge = (peak_tops * 1e12) / (bandwidth_gbps * 1e9)
    bound = "compute-bound" if intensity > ridge else "memory-bound"
    achievable = min(peak_tops * 1e12, intensity * bandwidth_gbps * 1e9)
    return intensity, ridge, bound, achievable / 1e9   # Gops/s attainable

# A depthwise 3x3 conv on a 56x56x64 feature map, int8 weights + activations
macs = 56 * 56 * 64 * 9
bytes_moved = (56*56*64) + (56*56*64) + (64*9)   # in + out + weights, 1 byte each
print(roofline(macs, bytes_moved, peak_tops=10, bandwidth_gbps=25))
# -> intensity ~8.9 ops/byte, ridge 400 ops/byte => memory-bound; ~223 Gops/s, far below 10 TOPS

The estimator above is the fastest sanity check in edge deployment. Before you trust a vendor's TOPS figure, run each dominant layer through it: if your model is mostly depthwise separable convolutions or attention over short sensor windows, it is probably memory-bound, and you should be shopping for bandwidth and on-chip SRAM, not raw TOPS.

import onnxruntime as ort

sess = ort.InferenceSession(
    "sensor_model.onnx",
    providers=["NnapiExecutionProvider",   # NPU / DSP via Android NNAPI
               "XnnpackExecutionProvider",  # optimized CPU SIMD
               "CPUExecutionProvider"])      # guaranteed fallback
out = sess.run(None, {"input": window})

Matching the workload to the engine

Placement follows from three questions asked per workload: how often does it run, how big is it, and how regular is its compute? An always-on trigger that runs every few milliseconds must live on the most frugal engine that can hold it, usually a DSP or MCU, because its energy cost is duty cycle times per-inference cost, and the duty cycle is brutal. A heavy but occasional model (run the full detector only when the trigger fires) belongs on the NPU or GPU, which are efficient per operation but expensive to wake, so you want them asleep most of the time. Irregular or unsupported operators, custom pre-processing, control flow, rare layer types, stay on the CPU, whichever way the rest of the graph is partitioned. This is the wake-up cascade pattern: a tiny gatekeeper on cheap silicon decides whether to pay for the expensive engine, and it is the single most important architectural move in battery-powered sensing.

The same logic scales up. An automotive perception SoC keeps lidar and radar preprocessing on DSPs, runs the BEV detection network (Chapter 43) on a large NPU, and reserves the CPU cluster for the planning and fusion logic that is all branches and no dense math. An event-camera system (Chapter 46) may skip conventional engines entirely for a neuromorphic core whose sparse, event-driven compute has no equivalent on a GPU. In every case the partition is dictated by duty cycle and operator regularity, not by which chip has the biggest headline number.

What's Next

In Section 59.3, we turn from choosing engines to the envelope they all share: memory, power, and thermal limits. The roofline showed that bandwidth and on-chip SRAM often matter more than raw arithmetic; next we quantify the memory hierarchy that feeds these engines, the energy each data movement costs, and the thermal ceiling that quietly throttles a chip long before its datasheet peak is reached.