Low-latency, low-power edge advantages and limits

"They sold me a sensor that burns a milliwatt. Then they handed me the event stream and I discovered the milliwatt was the cheap part. The processor I bolted on to make sense of it ran hot enough to warm my coffee."

A Budget-Conscious AI Agent

Why the edge advantages are real: three mechanisms

What the modality actually offers reduces to three mechanisms, each with a physical cause. First, latency without an exposure window: because a pixel reports a level crossing the instant it happens, there is no shutter to open and no frame to integrate, so the sensor-side latency is set by analog circuit bandwidth (single-digit microseconds) rather than a frame period (tens of milliseconds). Second, activity-proportional energy: a pixel that sees no change costs almost nothing, so a static or sparse scene draws idle power in the milliwatt range, and, crucially, the downstream compute is proportional to event count rather than to pixel count times frame rate. Third, bandwidth compression at the source: transmitting only changes can cut the data volume leaving the sensor by one to two orders of magnitude on sparse scenes, which relieves the interconnect and memory traffic that dominate edge energy budgets.

Why the third mechanism matters more than the first two is a fact every edge engineer learns eventually: on modern silicon, moving a byte across a chip costs far more energy than computing on it. A rough rule of thumb from Horowitz's oft-cited energy survey is that a 32-bit DRAM access costs on the order of hundreds of times more than a 32-bit arithmetic operation. So a sensor that natively emits less data is attacking the expensive part of the budget. How this compounds with neuromorphic hardware (Section 46.5) is that a spiking processor like Loihi 2 also skips computation for the pixels that produced no events, so sparsity is exploited twice, once in transmission and once in inference. When both halves cooperate, the end-to-end energy per decision on a genuinely sparse task can fall by an order of magnitude against a frame camera plus a dense CNN.

Where the advantages evaporate: the downstream pipeline

What undoes the sensor-side win, most often, is the very thing Chapter 46 has spent five sections building. The moment you accumulate events into a voxel grid or time surface (Section 46.2) and feed a dense CNN or a recurrent vision transformer (Section 46.3), you have re-densified the stream and re-imposed a frame-like clock: the network runs on fixed time windows, on a GPU or NPU whose energy is set by its tensor size, and the sensor's asynchrony is gone. The 10 microsecond sensor now sits behind a 15 millisecond inference tick, and the end-to-end latency is dominated by the slowest stage, exactly as Chapter 3's pipeline reasoning predicts.

Why this is not a bug you can simply patch away is that dense deep learning is where the accuracy lives today. Genuinely asynchronous, event-by-event models (spiking networks, Section 46.4) preserve the latency and energy story but trail dense representations on hard benchmarks. So the practical engineer faces a real trilemma between latency, energy, and accuracy, and the honest position is that you rarely get all three. When the sensor advantage genuinely survives to the actuator is the narrow case where the downstream model is either itself sparse and asynchronous (neuromorphic silicon) or so small and simple (a threshold, a centroid tracker, a tiny classifier on a microcontroller, Chapter 61) that it keeps pace with the stream without densifying it.

The small budget below makes the tradeoff concrete: it composes a photon-to-action latency and an energy-per-decision estimate from per-stage numbers, so you can see which stage dominates before you commit hardware.

def path_budget(stages):
    """Compose an end-to-end sensing path from per-stage (latency_us, energy_uJ).

    stages : list of (name, latency_us, energy_uJ) for each stage photon->action.
    Latency is additive along the serial path; the SLOWEST stage caps the
    sustainable decision rate. Energy per decision is the sum.
    """
    total_latency = sum(l for _, l, _ in stages)
    total_energy = sum(e for _, _, e in stages)
    bottleneck = max(stages, key=lambda s: s[1])          # largest latency stage
    max_rate_hz = 1e6 / bottleneck[1]                     # serial pipeline, no overlap
    return total_latency, total_energy, bottleneck[0], max_rate_hz

# A densified event pipeline: fast sensor, slow dense backbone dominates.
dense = [("DVS pixel",       10,   0.1),
         ("voxel accumulate", 500,  20.0),
         ("CNN on NPU",       15000, 900.0)]
# A neuromorphic pipeline: sparsity preserved end to end.
neuro = [("DVS pixel",       10,   0.1),
         ("SNN on Loihi 2",  300,   8.0)]

for name, path in [("dense", dense), ("neuro", neuro)]:
    lat, en, bott, rate = path_budget(path)
    print(f"{name:5s}: {lat/1000:6.2f} ms  {en:7.1f} uJ  "
          f"bottleneck={bott:14s} max {rate:6.0f} Hz")

Choosing the sensor: a decision rule with numbers

What you want is a repeatable test rather than a vibe. Event vision earns its place when three conditions hold together. First, the scene is sparse: the fraction of pixels changing per unit time is low, so the bandwidth and energy compression is large; a busy, textured, information-dense scene defeats it. Second, the relevant dynamics are fast or high-dynamic-range: microsecond latency or 120 dB range solves a problem a frame camera genuinely cannot, rather than one it merely does adequately. Third, the downstream task is light enough to stay sparse, either running on neuromorphic silicon or as a small classical operator, so the sensor advantage is not immediately densified away.

Why all three must hold is that any one failing collapses the case. Sparse but slow, and a cheap 30 Hz camera wins on cost and tooling maturity. Fast but dense, and you pay peak-throughput silicon that erases the energy story. Fast and sparse but needing a heavyweight classifier for accuracy, and the CNN dominates the budget regardless of the sensor. How to test it rigorously is to measure end to end on your own scene, under a leakage-safe protocol (Chapter 5): report accuracy, p99 latency, and energy per decision for the full path against a frame-camera baseline on the same task and split. A sensor-only spec sheet comparison is not evidence, because it silently omits the stage that usually decides the outcome.

What's Next

In Chapter 47, we leave the visible and change modality entirely, turning to radio-frequency and wireless sensing. There the signal is not light but the channel state of ambient Wi-Fi, and the same edge questions return in a new form: how to infer human presence, activity, and even vital signs from the way bodies perturb radio waves, and how to do it without the cameras whose privacy cost RF sensing quietly sidesteps.