Incident handling

"The pipeline never went down. It stayed up, cheerful and green, and confidently served nonsense to forty thousand devices for six hours. I have since learned that a heartbeat is not the same thing as a pulse."

A Wiser AI Agent

What counts as an incident, and how bad is it

An incident is any deviation from expected service that warrants an urgent, tracked response. For a streaming inference service the failure modes are broader than "the process crashed," because a crashed process is the easy case: it is loud, and a supervisor restarts it. The dangerous incidents are the quiet ones where the service stays up and wrong. They cluster into four families. Infrastructure failures are the classic ones: a consumer lags, a queue backs up, a node dies. Data failures are upstream: a sensor drops out, a firmware update changes a unit from Celsius to Fahrenheit, a timestamp clock skews so windows misalign (Chapter 3). Model failures are the shift itself: drift outruns the detector, or calibration collapses so the confidence scores lie (Chapter 18). Learning failures are unique to online systems: a burst of mislabeled or adversarial data poisons the running update, and the model degrades itself, permanently, unless the update is reversible.

You triage by severity, and severity is a function of blast radius times harm, not of how alarming the log looks. Anchor it to your service-level objective. If the SLO promises that the served error rate stays below a target, the error budget is the slack you are allowed to spend:

A one-in-ten-thousand miscalibration on a step counter is a low-severity annoyance; the same rate on an automotive emergency-braking classifier (Chapter 44) is a page-everyone, stop-the-line event. Writing the severity ladder before the incident, so the on-call engineer is not negotiating with themselves at 3 a.m., is most of the work.

Guardrails: the machinery that makes incidents survivable

Good incident response is designed in before the incident, as a small set of guardrails that let you contain damage fast. Four earn their place in almost every streaming sensor service. The kill switch for online learning is the most important and the most specific to this chapter: a single flag that freezes weight updates while inference continues, so a poisoning event stops compounding the instant you notice it. Because the learner keeps mutating, you pair the switch with versioned model checkpoints written on a cadence, so "roll back the model" means restoring a known-good snapshot from before the bad data arrived rather than retraining from scratch. A fallback model gives you something to serve while the primary is frozen or rolling back: often the last frozen offline model, or a simpler, dumber, but robust classical estimator (a Kalman predictor from Chapter 9) whose failure modes you fully understand. Finally, a circuit breaker wraps the model call so that when health signals cross a threshold, the breaker trips, stops routing traffic to the suspect model, and diverts to the fallback automatically, without waiting for a human.

The circuit breaker is worth dwelling on because it is the pattern that converts a slow human response into a fast automatic one. It has three states. Closed is normal: requests flow to the primary model and the breaker counts failures, where a "failure" for a perception service is not just an exception but a health-signal breach such as the drift flag firing or confidence collapsing below a floor. When failures exceed a threshold the breaker opens: all traffic diverts to the fallback and the primary gets a rest. After a cooldown it goes half-open, letting a trickle of traffic through to test whether the primary has recovered, and either closes again or re-opens. This gives you graceful degradation, a served-but-simpler answer, instead of a hard outage, and it buys the on-call engineer time to diagnose the root cause (Chapter 67) without the fleet burning down meanwhile.

import time

class ModelCircuitBreaker:
    """Serve the primary model until health degrades, then fail over to a fallback."""
    def __init__(self, primary, fallback, fail_threshold=5, cooldown_s=30):
        self.primary, self.fallback = primary, fallback
        self.fail_threshold, self.cooldown_s = fail_threshold, cooldown_s
        self.state, self.failures, self.opened_at = "closed", 0, 0.0

    def predict(self, x, health_ok):
        now = time.monotonic()
        if self.state == "open" and now - self.opened_at >= self.cooldown_s:
            self.state = "half_open"          # probe: let one request test recovery
        if self.state == "open":
            return self.fallback(x), "fallback"

        y = self.primary(x)                   # closed or half_open: try the primary
        if health_ok(x, y):                   # drift flag clear, confidence above floor
            self.failures, self.state = 0, "closed"
            return y, "primary"
        self.failures += 1
        if self.state == "half_open" or self.failures >= self.fail_threshold:
            self.state, self.opened_at = "open", now   # trip the breaker
        return self.fallback(x), "fallback"

Listing 60.7.1 is deliberately minimal so the state logic is legible: the primary is served while healthy, a run of health breaches opens the breaker to the fallback, and a single half-open probe after the cooldown decides whether to recover. In a real service you would additionally emit a metric on every state transition, because the transitions themselves are your highest-signal alert.

import pybreaker

breaker = pybreaker.CircuitBreaker(fail_max=5, reset_timeout=30)

@breaker
def serve(x):
    y = primary(x)
    if not health_ok(x, y):
        raise pybreaker.CircuitBreakerError("unhealthy prediction")
    return y

try:
    y = serve(x)                 # breaker counts failures and trips automatically
except pybreaker.CircuitBreakerError:
    y = fallback(x)              # degrade gracefully while the primary is out

Closing the loop: postmortem, replay, and prevention

Containment ends the bleeding; it does not stop the next incident. The last phase is a blameless postmortem, an SRE practice that transplants cleanly to sensor AI. Blameless means the writeup targets the system and its gaps, not the engineer who was on-call, because a culture that punishes the messenger simply stops reporting incidents, and unreported incidents are the ones that grow. The postmortem answers four questions in order: what was the customer-visible impact, what was the root cause (traced with the methods of Chapter 67, not guessed), what did detection and response actually cost in wall-clock, and what concrete change stops this class of failure. That last item is a tracked action, not a wish.

The measurements that make a postmortem honest are the two clocks: mean time to detect and mean time to recover. MTTD is dominated by whether correctness was observable at all, which is why the key insight above insists on promoting prequential and drift signals into alerts; MTTR is dominated by whether your guardrails are automatic, which is why the circuit breaker and kill switch pay for themselves. The single most valuable prevention tool for a streaming system is a replay harness: keep a rolling archive of raw input, and when an incident happens, replay the exact bytes through a candidate fix in a shadow deployment to confirm it would have caught the event, all under the leakage-safe discipline of Chapter 5 so you are not fooling yourself by evaluating on the very data you tuned against. A fix you cannot replay against the incident is a hypothesis, not a fix.

What's Next

In Chapter 61, we shrink the whole picture onto a microcontroller. Every guardrail here assumed you had spare cycles for a fallback model, headroom for checkpoints, and a network path for a page; TinyML strips all three away. We turn to designing sensor intelligence for kilobytes of RAM and milliwatts of power, where the fallback is a lookup table, the rollback is a flash partition, and incident handling becomes a firmware problem solved before the device ever ships.