"They asked why I think at the sensor instead of in the cloud. Because by the time my thought reaches the cloud and returns, the moment it was about has already happened without me."
A Punctual AI Agent
Prerequisites
This section assumes you can read a signal as a stream of timed samples (Chapter 3) and that you have met at least one deployed sensor model, such as a wearable activity classifier (Chapter 26) or a condition monitor (Chapter 37). No new mathematics is introduced here; the back-of-envelope arithmetic uses only rates and unit conversions. The hardware vocabulary this section leans on (NPU, DSP, MCU) is defined in the very next section, and the privacy machinery it gestures at is built out in Chapter 64.
The Big Picture
Every other chapter in this part optimizes how a model runs on constrained hardware. This one answers the prior question: why put it there at all, when a data center holds effectively unlimited compute? The answer is not nostalgia for embedded systems. It is that a sensor lives inside a physical control loop with a hard deadline, produces far more raw data than any link can afford to move, and observes things people did not consent to broadcast. Latency, bandwidth, energy, privacy, and availability each independently push inference toward the sensor. When you finish this section you will be able to look at a sensing problem and predict, before writing a line of model code, whether it belongs on-device, and say precisely which of those five forces decided it.
The round trip has a deadline the network cannot meet
A sensor rarely measures for the sake of measuring. It measures inside a loop: perceive, decide, act, and the action changes what the next measurement sees. The loop has a deadline set by physics, not by a product manager. An airbag controller reading a crash accelerometer has roughly \(15\) to \(30\) milliseconds between impact and the instant the occupant reaches the wheel. A robot arm closing a force-feedback grasp needs a control update on the order of a millisecond or the object slips or crushes. These deadlines are the what: the maximum time from sample to actuation that keeps the system stable and safe.
Now price the cloud round trip against that budget. A packet from a device to a regional data center and back crosses, at best, a few thousand kilometers of fiber. Light in fiber travels at roughly \(2 \times 10^8\ \mathrm{m/s}\), so \(1000\ \mathrm{km}\) each way already costs about \(10\) ms of pure propagation, before the radio access network, queueing, TLS, and server-side scheduling add their own tens of milliseconds. Measured end to end, a mobile request to a cloud model commonly lands in the \(50\) to \(150\) ms range, with a long tail that occasionally spikes to seconds when a cell tower is congested. That tail is the killer: a control loop is only as safe as its worst update, not its median.
Key Insight
On-device inference is not chosen because it is faster on average; it is chosen because it is bounded. A local NPU gives you a latency you can put in a datasheet and a safety case: this model returns in under \(N\) milliseconds, every time, with no dependency on a radio link you do not control. A cloud call has a good median and an unbounded tail, and a physical control loop is designed against the tail. Determinism, not raw speed, is the deciding property.
Real-World Application: automotive pedestrian braking
An automatic emergency braking system fuses a forward camera and a radar (Chapter 44) to decide whether to brake for a pedestrian stepping off a curb. At \(50\ \mathrm{km/h}\) the car covers about \(14\) metres every second, so \(150\) ms of added cloud latency is more than two metres of extra stopping distance, the difference between a hard stop and a collision. No car maker ships this in the cloud, and no regulator would certify it there. The perception model runs on an in-vehicle SoC precisely so its latency is bounded and survives a dead cellular connection in a tunnel. This is the round-trip argument in its starkest form: the deadline is set by kinetic energy, and the network cannot negotiate with it.
Raw sensor data is too big and too expensive to ship
Suppose latency did not matter. A second force still pushes inference down to the device: the sheer volume of what sensors produce. Consider the data rates. A single automotive lidar emits on the order of \(1\) to \(2\) million points per second; a \(1080\)p camera at \(30\) frames per second is tens of megabytes per second raw; even a humble triaxial IMU sampled at \(1\ \mathrm{kHz}\) is \(3000\) floating-point numbers a second, per sensor, and a wrist wearable carries several. Multiply by a fleet and the arithmetic becomes brutal.
The point is that inference is a form of compression. A model turns a firehose of raw samples into a trickle of decisions: fall detected, bearing fault, stage 2, heart rate 72. Running that compression at the sensor means you transmit the trickle, not the firehose. Running it in the cloud means you transmit the firehose and throw almost all of it away after it arrives. Let \(R\) be the raw sensor rate and \(D\) the decision rate the model emits. The compression ratio
\[ \rho = \frac{R}{D} \]is routinely \(10^3\) to \(10^6\) for sensor workloads, and every factor of \(\rho\) you keep local is bandwidth and cloud-ingest cost you never pay. Energy follows the same logic and often dominates it: on battery-powered devices the radio, not the processor, is the thirstiest component. Transmitting a bit over a cellular link can cost hundreds to thousands of times the energy of the arithmetic that would have classified it on-device. A wearable that streams raw accelerometry to the cloud dies before lunch; one that classifies locally and uploads a daily summary runs for a week on the same battery.
Field Story: a vibration monitor on a remote pump
An industrial predictive-maintenance node (Chapter 36) sits on a water pump in a field with only a low-power LTE-M radio and a small solar panel. The accelerometer runs at \(25.6\ \mathrm{kHz}\) to catch high-frequency bearing tones, which is roughly \(100\) kB of raw data per second, far more than the radio or the energy budget can sustain around the clock. The engineers put a small spectral-feature model (Chapter 7) on the node. It processes the full-rate stream locally and transmits only a fault probability and a handful of band energies once a minute, a few dozen bytes. The node now runs indefinitely on solar, and the backhaul carries decisions instead of a raw signal nobody would ever have the bandwidth to read.
You can make this concrete in a few lines. The snippet below compares the yearly bytes leaving a device under two policies: stream every raw sample, or run the model locally and emit one decision per second.
raw_hz = 25_600 # accelerometer samples per second
bytes_each = 4 # float32 per sample
decision_hz = 1 # model emits 1 result/second on-device
decision_b = 16 # bytes per decision (probability + a few bands)
sec_per_year = 365 * 24 * 3600
stream_gb = raw_hz * bytes_each * sec_per_year / 1e9
ondevice_gb = decision_hz * decision_b * sec_per_year / 1e9
print(f"stream raw: {stream_gb:8.1f} GB/year")
print(f"on-device: {ondevice_gb:8.4f} GB/year")
print(f"ratio: {stream_gb / ondevice_gb:,.0f}x less data moved")As Listing 59.1.1 makes brutally clear, the choice is not marginal. The gap between shipping raw data and shipping decisions is measured in orders of magnitude, and that gap is what pays for the whole edge-AI discipline that fills this part of the book.
The Right Tool
You do not hand-write the on-device inference runtime. Converting a trained model into a form that runs in kilobytes of RAM on a phone or microcontroller is a solved problem: LiteRT (the runtime formerly shipped as TensorFlow Lite) or ExecuTorch will take a trained graph, quantize it, and produce a self-contained interpreter call.
import tensorflow as tf
conv = tf.lite.TFLiteConverter.from_saved_model("pump_model")
conv.optimizations = [tf.lite.Optimize.DEFAULT] # int8 quantization
open("pump_model.tflite", "wb").write(conv.convert())convert() call. What quantization actually does to the weights is the subject of Section 59.4.Listing 59.1.2 removes the plumbing, not the judgment: whether int8 keeps your accuracy is an evaluation question you still own, and one this part treats with care.
Privacy, autonomy, and availability finish the argument
Three further forces settle cases that latency and bandwidth alone leave ambiguous. The first is privacy. Sensor data is intimate in a way that clicks and text are not: a microphone hears everything in a kitchen, a camera sees who is home, PPG and ECG streams are biometric identifiers and health data under regulations like GDPR and HIPAA (Chapter 34). Data that never leaves the device cannot be breached in transit, subpoenaed from a server, or repurposed by a vendor. Keeping raw biosignals on the wrist and shipping only derived, non-identifying summaries is often the only design a regulator or a hospital will accept, which is why on-device inference is the foundation the federated methods of Chapter 64 build on.
The second is availability. A sensor system that stops working when the network does is not a product; it is a demo. A pacemaker, a factory safety interlock, a drone navigating a canyon, a car in a tunnel: all must keep perceiving through a dead link. On-device inference makes correct behaviour the default state rather than the connected state. The third is cost and autonomy at fleet scale: a million devices each making a cloud call per second is a per-inference bill and a server farm you must operate and scale; the same million devices inferring locally cost nothing per decision after manufacture. The economics invert completely once the fleet is large.
Research Frontier
The frontier is pushing full foundation-scale perception onto the device, not just small classifiers. Apple's on-device foundation models (2024, roughly \(3\) billion parameters running on-phone) and Google's Gemini Nano, together with the MobileNetV4 (2024) and EfficientViT vision backbones and MCUNet-style tiny networks (Lin et al., NeurIPS 2020), show sensor-reasoning and vision-language capability (Chapter 21) shrinking into phone and even microcontroller budgets. The open question this part keeps circling is how far the five forces of this section can be satisfied at once: how much capability survives when latency must be bounded, energy is a battery, and the weights must fit in on-chip memory. The rest of Chapter 59 is the toolkit for answering that.
None of this abolishes the cloud. Training happens there, large models are served there, and fleets report summaries there. The professional stance is a spectrum, not a religion: some computation belongs at the sensor, some in the cloud, and the interesting engineering is where you draw the line. Deciding that split, the edge-cloud partition, is important enough that it gets its own treatment in Section 59.7. This section only insists on the prior point: the default gravity for sensory AI pulls toward the device, because the sensor is where the deadline, the data, and the privacy all physically live.
Exercise
Pick a sensing product you use: a smartwatch, a video doorbell, a car, a robot vacuum. For each of the five forces in this section (latency, bandwidth, energy, privacy, availability), write one sentence stating whether it pushes that product's core perception on-device or leaves it comfortably in the cloud. Then name the single force you believe was decisive, and one piece of evidence from the product's observable behaviour (battery life, offline behaviour, what it uploads) that supports your guess.
Self-Check
1. Why is a cloud model's median latency the wrong number to design a physical control loop against, and what number should you use instead?
2. Restate "inference is a form of compression" using the ratio \(\rho = R/D\). Why does a large \(\rho\) argue for computing at the sensor rather than in the data center?
3. On a battery device, transmitting a bit often costs far more energy than computing on it. What design conclusion follows for a wearable that must run all day?
What's Next
In Section 59.2, we open the box the model has to run inside. Having established why inference belongs at the sensor, we survey the silicon that hosts it: the tradeoffs between CPU, GPU, NPU, DSP, and microcontroller, and how each choice reshapes the latency, energy, and memory budgets this section treated as fixed constraints.