Latency and real-time constraints

"My perception is flawless. It only describes a world that stopped existing forty milliseconds ago."

A Punctual AI Agent

This section assumes the timestamping and clock-synchronization machinery of Chapter 3: without a common, monotonic clock across sensors and compute, none of the latency accounting below is even measurable. It also builds on the delayed-measurement problem raised at the end of Section 57.4, where an exteroceptive fix for time \(t\) arrives long after \(t\). Here we treat delay as a first-class system property to be budgeted, measured, and compensated, not a nuisance to be ignored.

The sensor-to-action latency budget

The quantity that matters is not the frame rate of any one sensor but the end-to-end latency \(\tau\): the wall-clock interval from a physical event to the actuator command it influences. Throughput and latency are different axes. A pipeline can produce 60 fused poses per second (high throughput) while each pose is 120 ms old by the time it is published (high latency), because deep pipelines hide latency inside parallel stages. A perception system is fast in the sense that matters only when \(\tau\) is small, and \(\tau\) is the sum along the critical path, not the maximum stage.

Why a budget and not just "as fast as possible"? Because latency in a feedback loop is a phase lag, and phase lag destroys stability. A controller that reacts to a state delayed by \(\tau\) is applying a correction computed for where the robot was, and if \(\tau\) grows past what the control gains assume, the loop overshoots and can diverge. A first-order feel for the ceiling: a delay of \(\tau\) contributes phase lag \(\phi = \omega\tau\) at angular frequency \(\omega\), so at the loop's crossover frequency \(\omega_c\) you are spending phase margin

and once that consumes the available margin the system rings or goes unstable. This is why a legged robot's balance loop budgets single-digit milliseconds while a warehouse mobile base tolerates tens, and why "add a bigger network for more accuracy" is not free: every extra millisecond of inference is spent from a fixed phase-margin account.

Where the milliseconds go

Total latency decomposes into stages, each with its own typical cost and its own tail. Sensor exposure and readout is often overlooked: a rolling-shutter camera finishes reading its last row several milliseconds after its first, and a spinning lidar's points for one revolution span a full rotation period. Transport covers the bus and driver: USB, Ethernet, or a CAN frame, plus the operating-system queue where a packet waits for a scheduler slot. Compute is the neural network and the classical fusion, and it is where a garbage-collection pause, a cache miss, or thermal throttling injects the long tail. Queueing and serialization between processes, especially across the message bus of Section 57.7, adds delay that grows nonlinearly once any stage saturates and back-pressure builds, the same failure studied for streaming systems in Chapter 60.

The only way to manage this is to measure it, per stage, under load, and read the percentiles rather than the mean. The snippet below simulates a 30 Hz depth-perception pipeline, gives each stage a base cost plus a heavy-tailed spike, and reports where the end-to-end distribution lands against a one-frame deadline.

import numpy as np
rng = np.random.default_rng(0)

def stage(base, jitter, tail_p, tail_add, n):
    """One pipeline stage: Gaussian body plus a rare exponential tail (ms)."""
    x = rng.normal(base, jitter, n)
    spikes = rng.random(n) < tail_p          # occasional GC / throttle / cache miss
    x[spikes] += rng.exponential(tail_add, spikes.sum())
    return np.clip(x, 0.1, None)

n = 100_000
exposure  = stage(5,  0.3, 0.000, 0,  n)     # rolling-shutter + readout
transport = stage(2,  0.5, 0.010, 8,  n)     # bus + driver queue
inference = stage(18, 1.5, 0.020, 25, n)     # the neural net, occasional throttle
postproc  = stage(3,  0.4, 0.005, 5,  n)     # tracking + fusion
total = exposure + transport + inference + postproc

deadline = 33.3                               # one 30 Hz frame period (ms)
for name, x in [("exposure", exposure), ("transport", transport),
                ("inference", inference), ("postproc", postproc),
                ("END-TO-END", total)]:
    print(f"{name:11s} mean={x.mean():5.1f}  p50={np.percentile(x,50):5.1f}"
          f"  p99={np.percentile(x,99):6.1f}  p99.9={np.percentile(x,99.9):6.1f}")

print(f"\ndeadline={deadline} ms   frames over budget: "
      f"{100*(total > deadline).mean():.2f}%")

Run it and the lesson is immediate: the mean clears the deadline with room to spare, but the p99.9 does not, and a few percent of frames arrive late. Cutting the inference tail (quantization, a smaller model, an accelerator) is the highest-leverage fix, which is the whole subject of edge model optimization in Chapter 59.

Hard, firm, and soft real time

"Real-time" does not mean "fast"; it means bounded and predictable. A system is real-time when it guarantees a response before a stated deadline, and the value of a late result defines the class. A hard deadline means a late result is a failure with possibly catastrophic cost (a balance controller, an emergency-brake trigger). A firm deadline means a late result is simply discarded as useless (a perception frame that missed its control cycle). A soft deadline means a late result still has decaying value (a mapping update, a telemetry log). Most robot perception is firm-to-soft; the control loop it feeds is hard. Knowing which is which tells you whether to fight for the tail or to drop the frame and move on.

Meeting a hard deadline requires determinism, and determinism is a property of the whole stack, not the algorithm. Worst-case execution time (WCET), not average time, is the design number. General-purpose operating systems, dynamic memory allocation, page faults, and garbage collection all inject unbounded jitter, which is why hard real-time control runs on an RTOS or a `PREEMPT_RT` kernel with priority-based preemptive scheduling, memory locked into RAM, and no allocation on the hot path. A subtle killer here is priority inversion: a high-priority perception task blocked on a mutex held by a low-priority task that a medium task has preempted, so the urgent work waits on the least urgent. Priority inheritance protocols exist precisely to bound this. The practical rule is to keep the hard-real-time control loop small, deterministic, and isolated, and to let the heavier, jittery perception run at firm deadlines feeding it, never the other way around.

Living with latency: timestamp, compensate, predict

You cannot eliminate \(\tau\), so you account for it. Three disciplines make a delayed estimate usable. First, timestamp at the source: every measurement carries the capture time, not the arrival time, so downstream code always knows how old the data is and can associate it with the correct control cycle. Second, handle out-of-sequence measurements: a lidar pose for time \(t\) that lands 80 ms late must be fused against the state as it was at \(t\) (via a short state buffer, then re-propagated to now), not slammed into the current state, a correction that the smoothing view of Chapter 11 formalizes. Third, predict forward to hide the residual delay: use the high-rate proprioceptive process model to extrapolate the last good estimate to the present instant, so the controller acts on a prediction of now rather than a measurement of then. Latency compensation is why fast proprioception (Section 57.1) is worth so much: it is the only signal fresh enough to paper over the lag of everything slower.

# Record an instrumented run, then analyze the latency of a message chain.
ros2 trace start perception_session
ros2 launch my_robot perception.launch.py      # exercise under real load
ros2 trace stop perception_session
# -> Jupyter analysis yields per-callback and end-to-end p50/p99/p99.9 tables

Whether you compensate by prediction or by re-propagation, the honest move is to propagate the age of the data into the uncertainty: an estimate extrapolated 60 ms forward is less certain than a fresh one, and a controller that knows this can slow down rather than commit to a stale guess. That coupling of latency to reported confidence is what keeps a punctual-but-wrong perception system from being worse than a slow-but-honest one.

What's Next

In Section 57.7, we bring the whole chapter together on a concrete platform: integrating the perception stack in ROS 2, where nodes, topics, quality-of-service settings, TF frame trees, and executors turn the timestamping, fusion, and latency-compensation principles of this section into a running system you can launch, trace, and deploy.