Backpressure and data loss

"The sensor never asked whether I was ready. It just kept sending, and every sample I dropped was a small decision I would rather have made on purpose."

An Overwhelmed AI Agent

This section assumes you understand sampling rates and timestamps from Chapter 3, and the compute and memory limits of edge hardware from Chapter 59. The stateful-window machinery of Section 60.1 is the consumer we are trying to protect. We speak in terms of a producer (the sensor and its driver), a bounded queue, and a consumer (feature extraction plus inference).

Backpressure: the arithmetic of a rate mismatch

Backpressure is the signal a slow consumer sends upstream to say "stop, I am full." It is the load-regulating mechanism of every well-behaved pipeline. The clean way to see when it is needed is a queue balance. Let the producer emit samples at mean rate \(\lambda\) (samples per second) and the consumer drain them at mean service rate \(\mu\). The queue occupancy \(L\) obeys, on average, the flow relation from Little's Law:

$$ L = \lambda \, W, \qquad \rho = \frac{\lambda}{\mu}, $$

where \(W\) is the mean time a sample waits in the system and \(\rho\) is the utilization. When \(\rho < 1\) the queue is stable and \(W\) is finite. When \(\rho \ge 1\), even momentarily, the queue grows without bound and waiting time diverges. The essential fact of streaming sensor AI is that \(\lambda\) is set by the physical world and is often constant, while \(\mu\) is not: a heavier input (more objects in a radar frame, a burst of events), a thermal throttle on the CPU, or a garbage-collection pause can all drop \(\mu\) below \(\lambda\) for a stretch. During that stretch, backpressure is the only thing standing between you and an out-of-memory crash.

Bounded queues and drop policies

The single most important architectural rule is: every queue between the sensor and the model must be bounded. An unbounded queue does not prevent data loss, it merely defers and disguises it, converting a small, controlled drop rate now into a catastrophic latency spike and a process death later. Once the queue is bounded at capacity \(C\), you must specify what happens on overflow. There are four canonical policies:

• Block (apply true backpressure). The producer stalls until space frees up. Correct only when the producer can wait, for example a file replay or a network source. Fatal for a live sensor whose driver will overflow its own hardware FIFO while you block, moving the loss somewhere you cannot see or measure.
• Drop-newest (tail drop). Reject the incoming sample when full. Simple, but it discards the freshest data, which is usually the most valuable for real-time inference.
• Drop-oldest (head drop). Evict the stalest queued sample to admit the new one. This keeps the pipeline current and is the right default for most latency-sensitive perception, because a stale detection is often worse than useless.
• Decimate or sample. Keep every \(k\)-th sample, or use reservoir sampling to retain a statistically representative subset. This is the graceful-degradation option: you deliberately lower the effective rate rather than losing contiguous chunks.

The choice is not cosmetic. For a fall detector, drop-oldest preserves the moment of impact; for a vibration-spectrum monitor, contiguous drops corrupt the FFT window (Section 60.1), so decimation that preserves uniform spacing is far safer. Match the policy to what your downstream feature extraction assumes about temporal continuity.

Measuring loss, and choosing what to shed

You cannot manage what you do not measure. A production streaming service must export at least three quantities continuously: queue depth (how full is the buffer, a leading indicator of trouble), drop rate (samples discarded per second, and cumulatively), and end-to-end latency or staleness (the age of the sample currently being processed). Watch queue depth against its capacity: a buffer that sits near-empty then spikes is telling you \(\mu\) is bursty; one that rides permanently near capacity is telling you \(\rho \approx 1\) and you are one hiccup from sustained loss. These signals are also the raw material for Section 60.7 on incident handling and for fleet-wide health monitoring in Chapter 69.

The mini-service below implements a bounded queue with a drop-oldest policy and the three metrics, wrapping a producer thread that mimics a fixed-rate sensor and a consumer whose service time occasionally spikes.

import collections, threading, time

class BoundedStream:
    def __init__(self, capacity):
        self.q = collections.deque(maxlen=capacity)  # deque auto-evicts oldest
        self.lock = threading.Lock()
        self.produced = self.dropped = 0

    def push(self, sample):                          # producer side (sensor)
        with self.lock:
            self.produced += 1
            if len(self.q) == self.q.maxlen:         # full: drop-oldest
                self.dropped += 1                    # the evicted sample is lost
            self.q.append(sample)                    # maxlen deque drops head for us

    def pop(self):                                   # consumer side (inference)
        with self.lock:
            return self.q.popleft() if self.q else None

    def health(self):
        with self.lock:
            depth = len(self.q)
            loss = self.dropped / max(self.produced, 1)
            return {"depth": depth, "cap": self.q.maxlen, "drop_rate": round(loss, 3)}

stream = BoundedStream(capacity=4)
for t in range(10):                                  # a burst the consumer cannot match
    stream.push({"t": t})
print(stream.health())   # -> {'depth': 4, 'cap': 4, 'drop_rate': 0.6}: 6 of 10 shed on purpose

Beyond queue policy, two higher-level moves shed load intelligently. Adaptive quality swaps in a cheaper model (a smaller network, a coarser stride) when queue depth crosses a high-water mark, and restores the full model when it recovers, trading accuracy for throughput exactly when \(\rho\) demands it. Uncertainty-aware shedding spends compute where it matters: skip frames the model is already confident about and reserve full inference for ambiguous ones, using the calibrated confidence machinery of Chapter 18. Both keep \(\mu\) above \(\lambda\) without a blind, uniform sacrifice of data.

import asyncio
q = asyncio.Queue(maxsize=4)         # bounded: put() blocks when full
# producers await q.put(x); consumers await q.get(); backpressure is built in

What's Next

In Section 60.3, we assume the flow-control battle is won and the samples that survive are arriving at a manageable rate. The next problem is computing features and normalization statistics incrementally, one sample at a time, without ever storing the whole stream, so that means, variances, and scalings stay correct even as the data itself keeps moving.