Reliability-engineering patterns

"My model was 99.9% accurate. The wiring harness, the NTP daemon, and the retry loop I forgot to bound were not."

A Humbled AI Agent

A note on prerequisites. This section leans on the redundancy and voting structures of Section 68.3 and the innovation-consistency checks you first met with the Kalman family in Chapter 9. The quantitative reliability arithmetic reuses the basic probability of Chapter 4, and the remaining-useful-life view of a degrading component connects directly to the prognostics of Chapter 36.

Quantifying reliability: hazard, MTBF, and availability

You cannot budget what you cannot measure, so reliability engineering starts with a small set of definitions. Model a component's time-to-failure as a random variable \(T\) with survival function \(R(t)=P(T>t)\), the reliability. The hazard rate \(\lambda(t)\) is the instantaneous failure rate given survival so far, \(\lambda(t)=f(t)/R(t)\), and it traces the familiar bathtub curve: high infant-mortality early, a low flat useful-life plateau, and a rising wear-out tail. On the flat plateau failures are approximately Poisson with constant \(\lambda\), giving the exponential model \(R(t)=e^{-\lambda t}\) and a mean time between failures \(\text{MTBF}=1/\lambda\). What operators actually sell against is availability, the fraction of time the system is up,

where MTTR is the mean time to repair or recover. The lesson hidden in that ratio is that recovery speed matters as much as failure frequency: a sensor node that fails weekly but self-recovers in two seconds is more available than one that fails yearly but needs a technician. This is why so many of the patterns below are about shrinking MTTR (fast detection, automatic restart, degraded-but-alive operation) rather than chasing an unreachable zero failure rate.

Composing blocks: series, parallel, and k-out-of-n

Real systems are built from many components, and a reliability block diagram (RBD) tells you how their reliabilities combine. Components in series (all must work: the lens, the image sensor, the ribbon cable, the SoC) multiply, \(R_\text{series}=\prod_i R_i\), so a long series chain is always weaker than its weakest link and adding parts can only hurt. Components in parallel (any one suffices: three redundant IMUs feeding a voter) combine through their failure probabilities, \(R_\text{parallel}=1-\prod_i(1-R_i)\). The general redundancy structure from Section 68.3 is k-out-of-n: the system works if at least \(k\) of \(n\) identical units work, with

Two facts from this arithmetic drive most architecture decisions. First, redundancy only helps if the units fail independently; a common-cause failure (shared power rail, shared clock, the same rain on all three cameras, the same bug in one model replicated three times) collapses parallel blocks back into a single block, which is why 68.3 insisted on diverse redundancy. Second, adding a low-reliability component in series with a strong one wastes the strong one. The code below evaluates an RBD for a triple-modality front end so you can see how a single fragile series element caps the whole system.

import numpy as np
from math import comb

def R_series(rs):            # all must survive
    return np.prod(rs)

def R_parallel(rs):          # any one survives
    return 1.0 - np.prod([1.0 - r for r in rs])

def R_k_of_n(r, n, k):       # at least k of n identical units
    return sum(comb(n, i) * r**i * (1 - r)**(n - i) for i in range(k, n + 1))

# A perception front end: a shared power supply (series) feeds a
# 2-of-3 redundant sensing tier, which feeds a shared inference SoC (series).
R_power = 0.999            # common-cause: NOT redundant
R_sensor = 0.98           # one sensing modality over the mission
R_soc = 0.995

R_sensing_tier = R_k_of_n(R_sensor, n=3, k=2)   # tolerates one dead modality
R_system = R_series([R_power, R_sensing_tier, R_soc])

print(f"2-of-3 sensing tier reliability: {R_sensing_tier:.5f}")
print(f"end-to-end system reliability:   {R_system:.5f}")
print(f"limiting factor is the shared power rail (R={R_power})")

As the caption notes, the redundant tier reaches roughly 0.9988 on its own, yet the system tracks the shared power rail. That is the RBD earning its keep: it points your reliability budget at the single points of failure instead of at the part you were already proud of.

Structural patterns: degrade, isolate, and time-bound

Beyond the arithmetic sits a catalogue of structural patterns, each a reusable answer to a recurring failure shape. Graceful degradation defines an explicit ladder of reduced-capability modes so the system steps down instead of falling over: a fusion stack that loses lidar drops to a camera-plus-radar mode with a lower speed limit rather than blanking out, an idea that connects directly to the missing-modality robustness of Chapter 50. The related distinction is fail-operational versus fail-safe: a parked robot arm can fail safe (stop and hold), but a vehicle at highway speed must fail operational (keep perceiving well enough to reach a safe state), which forces genuine redundancy rather than a mere shutoff.

The bulkhead pattern isolates resources so one flooding subsystem cannot sink the ship: give the safety-critical obstacle detector its own CPU budget, memory pool, and thread so a runaway logging task or a slow map download cannot starve it. The circuit breaker wraps a call to an unreliable dependency (a cloud model, a remote map tile server, a neighbor node) and, after a threshold of failures, trips open to return a fast cached or degraded answer instead of piling up timeouts, then probes periodically to reset. Every remote or retry-able call needs a bounded timeout with backoff; an unbounded retry loop is a latent outage, because the moment the dependency stalls, your real-time loop stalls with it. Finally, the watchdog and heartbeat pair is the oldest embedded pattern of all: a task that must periodically kick a hardware timer or emit a heartbeat, so that a hung inference thread is detected within one watchdog period and the node is restarted, shrinking MTTR from minutes to milliseconds.

Analysis and budgets: FMEA, fault trees, and error budgets

Patterns are chosen, not sprinkled, and two classic analyses tell you which failures to defend against. Failure modes and effects analysis (FMEA) walks bottom-up through every component, lists how it can fail, and scores each mode by a risk priority number, the product of severity, occurrence, and detectability, \(\text{RPN}=S\times O\times D\); the high-RPN rows earn a pattern. Fault-tree analysis (FTA) works top-down from a hazard (for example, "vehicle fails to brake for a real obstacle") through AND and OR gates to the combinations of component faults that cause it, and its minimal cut sets are exactly the single points of failure and common-cause pairs your RBD must not contain. These feed the ISO 26262 and SOTIF work of Section 68.5 and the argument of Section 68.6, but they are reliability tools first. On the operational side, the SRE error budget reframes reliability as a resource to spend: if the availability target is 99.9%, you are permitted about 43 minutes of downtime a month, and you spend it deliberately on deployments and experiments rather than pretending zero downtime is free. Error budgets pair naturally with the drift and shift monitoring of Chapter 66, because a model degrading under distribution shift burns availability just as surely as a crashed process.

What's Next

In Chapter 69, we take these reliability patterns off the single device and out to the whole fleet: data contracts that keep the sensor stream trustworthy, model versioning and staged deployment, drift and data-quality monitoring under delayed ground truth, retraining triggers, and the incident response that turns a monitor's alert into a bounded, auditable recovery. The availability arithmetic and the watchdog, circuit-breaker, and error-budget patterns you just met become the operational backbone of running thousands of sensing nodes in production.