"I could not prove the perception network was correct, so I stopped trying. Instead I gave it a much simpler colleague whose only job was to notice, in real time, when it had clearly stopped being correct."
A Pragmatic AI Agent
Prerequisites
This section builds on the fault taxonomy of Section 68.1 and the redundancy and safety-envelope patterns of Section 68.3; a monitor is what enforces an envelope and what arbitrates a vote. It leans on the change-detection statistics of Chapter 12, the calibrated uncertainty and conformal guarantees of Chapter 18, and the out-of-distribution detectors of Chapter 66. We assume a trained perception model exists and may be wrong in ways we cannot fully enumerate in advance. This section is about the machinery that watches it while it runs, not about the standards that demand such machinery (that is Section 68.5) or the argument that it is sufficient (Section 68.6).
The Big Picture
You cannot exhaustively verify a deep perception network before deployment: its input space is astronomically large and its failure surface is not something a proof will close. Runtime assurance sidesteps that impossibility. Instead of certifying the complex component is always right, you deploy a second, far simpler component whose only job is to detect, online and with bounded latency, that the complex one has left its region of validity, and then to trigger a safe response. The complex network is the doer; the simple monitor is the checker. Because the checker is small, its own correctness is tractable to argue. This asymmetry, hard to build, easy to check, is the load-bearing idea of the whole section. Monitors watch four surfaces (inputs, internal state, outputs, and timing), and their alarms feed a decision layer that degrades gracefully rather than failing silently.
Doers, checkers, and the Simplex architecture
The canonical pattern is the Simplex architecture (Sha, 2001) and its avionics descendant, the Run-Time Assurance (RTA) switch used in modern autonomy stacks. Three parts: a high-performance untrusted controller (the learned perception-and-planning stack), a simple trusted safety controller (a conservative fallback), and a decision module (the monitor) that switches from the former to the latter the instant the system approaches the boundary of a certified-safe operating region. The entire safety argument rests not on the untrusted stack but on two much weaker claims: the monitor detects boundary approach in time, and the fallback keeps the system safe once engaged. Both are simple enough to verify by conventional means.
The reason this works is a separation of concerns. Correctness of a perception network is a property over an intractable input distribution. Safety of the fallback is a property over a small, hand-designed controller. Detectability of a boundary crossing is a property of a monitor you also designed to be simple. You have traded one impossible obligation for three feasible ones. The cost is that the fallback is conservative, so every false alarm from the monitor degrades performance; monitor design is therefore a tuning of the miss-rate versus false-alarm-rate tradeoff you met in Chapter 12, now with asymmetric, safety-weighted costs.
Key Insight
A monitor is only as trustworthy as its independence from the thing it monitors. If your out-of-distribution check is a head bolted onto the same backbone that produces the perception output, a corrupted feature map fools both at once and the monitor rubber-stamps the failure. Effective monitors are built on a different information path: a physics-based plausibility model, a redundant sensor of a different modality, an analytically simple bound, or a separately trained detector on raw inputs. Diversity is not a nicety here; a monitor that shares a failure mode with the doer provides assurance only on paper. This is the same argument that makes cross-modal voting in Section 68.3 stronger than replicating one channel three times.
What to watch: inputs, internals, outputs, and timing
A complete runtime-monitoring layer instruments four surfaces, because failures announce themselves on different ones.
- Input monitors ask whether the current sensor frame is one the model has any business processing. This is online out-of-distribution and sensor-health detection: density or reconstruction scores on raw inputs, coverage checks against the training manifold (Chapter 66), and the stuck-at, dropout, and saturation flags from Section 68.1. An input the network never saw in training gets no meaningful output, so catching it early is the cheapest save available.
- Internal monitors read the model's own confidence. A calibrated predictive-uncertainty signal or a conformal nonconformity score (Chapter 18) that exceeds its validation quantile is a self-reported "I am guessing." Conformal prediction is especially valuable at runtime because its abstention rate carries a distribution-free statistical guarantee, turning a soft confidence into a monitorable threshold.
- Output monitors ask whether the produced estimate is physically and logically plausible: does the fused object velocity exceed anything a car can do, does an estimated depth violate the ground plane, does a tracked target teleport between frames? These are cheap invariants derived from physics and geometry, and they catch confident-but-wrong outputs that input and internal monitors miss.
- Timing and liveness monitors are the watchdogs: heartbeat counters, end-to-end latency budgets, and staleness checks. A perception result that is correct but 300 ms late is a safety fault, and a hung inference thread must be caught by a timer, not by inspecting its (never-arriving) output.
No single surface is sufficient. A spoofing attack (Section 68.2) may produce an in-distribution input, a confident model, and a plausible output, and be caught only by cross-checking against a redundant modality. The layered set is what gives coverage.
Specifying monitors: temporal logic and robustness
Ad-hoc if statements do not scale to a safety case, so the mature practice is to write monitored properties as formal specifications and synthesize the checker from them. Signal Temporal Logic (STL) is the workhorse: it expresses properties over real-valued signals with bounded-time temporal operators. "The estimated headway must always, over the next horizon \(H\), stay above the braking-distance envelope" is written
where \(\square\) is the "always" operator. STL comes with a quantitative semantics: instead of a bare true/false, a monitor computes a robustness degree \(\rho(\varphi, s, t)\) whose sign gives satisfaction and whose magnitude gives the margin. For the property above,
$$ \rho(\varphi, s, t) \;=\; \min_{t' \in [t, t+H]} \bigl(d_{t'} - d_{\text{safe}}(v_{t'})\bigr). $$A positive robustness with a comfortable margin means "safe with room"; a robustness trending toward zero is an early warning the doer is eroding its safety envelope before any hard violation occurs. That gradient is exactly the boundary-approach signal the Simplex decision module needs, which is why STL robustness has become the standard interface between perception monitors and RTA switches.
import numpy as np
def d_safe(v, reaction=1.0, decel=6.0, margin=2.0):
"""Braking-distance envelope: reaction gap + kinematic stop + fixed margin."""
return v * reaction + v**2 / (2 * decel) + margin
def stl_always_robustness(dist, vel, horizon):
"""Robustness of G_[0,H] (dist >= d_safe(vel)) at each timestep.
Positive = satisfied with margin; the min over the horizon is the guarantee."""
slack = dist - d_safe(vel) # per-step signed margin
rho = np.full_like(slack, np.nan)
for t in range(len(slack) - horizon):
rho[t] = np.min(slack[t : t + horizon + 1]) # 'always' = worst case ahead
return rho
# A run where a late, confident perception glitch shrinks the headway.
vel = np.full(60, 18.0) # 18 m/s (~65 km/h)
dist = np.concatenate([np.full(40, 55.0), np.linspace(55, 34, 20)])
rho = stl_always_robustness(dist, vel, horizon=10)
warn = np.nanargmax(rho < 5.0) # first step the margin drops below 5 m
brk = np.nanargmax(rho < 0.0) # first step the envelope is actually violated
print(f"margin at t=0: {rho[0]:.1f} m | early-warning at t={warn} | violation at t={brk}")
print(f"warning budget: {brk - warn} steps before the fallback must be engaged")The code above turns the specification \(\square_{[0,H]}(d \ge d_{\text{safe}})\) into a running scalar. Notice it never inspects the perception network's weights; it consumes only the network's output and a physics model, keeping the monitor on an independent information path as the Key Insight demanded. Reading the robustness trend, not just its sign, is what buys the early warning.
Library Shortcut
The hand-rolled loop above handles a single "always" operator. Real specifications nest "eventually," "until," and time bounds, and computing their robustness efficiently over streaming data is fiddly (the min/max envelope needs a sliding-window monotonic-deque trick to stay linear-time). The rtamt library parses full STL from a string and gives you an online, incremental monitor: spec.declare_var('d','float'); spec.spec = 'always[0,10](d >= dsafe)'; spec.parse() then spec.update(...) per sample. Roughly 120 lines of careful window-management code collapse to about 6, and you get correct bounded-future and past-time operators for free. Use the hand-written version to understand robustness; use rtamt in anything that ships.
Practical Example: a highway lane-keeping fallback
An automotive Level-2 stack runs a learned lane-and-object perception model at 30 Hz. Its RTA monitor is deliberately dumb: an STL bank checks headway margin (the property above), an input monitor flags when raw camera entropy collapses (low-sun glare, tunnel entry) or the radar returns disagree with vision beyond a gate, and a 40 ms watchdog guards the inference thread. During a sudden sun-glare transition at a hill crest, the vision confidence stays high (the network hallucinates a clear lane) but the radar-vision cross-check robustness goes negative within 60 ms. The monitor does not try to fix perception; it hands control to the trusted fallback, a simple controller that holds the last confident heading and commands a gentle deceleration, and raises a driver takeover request. Three frames later the glare clears, the cross-check recovers positive margin, and the RTA switch releases control back. The learned stack was wrong for 90 ms; the monitor made those 90 ms uneventful. This is the whole value proposition: assurance without solving perception.
From alarm to safe action: latency, coverage, and the case
A monitor that detects a hazard too late has detected nothing useful. The governing quantity is the reaction-time budget: the physical time-to-hazard must exceed detection latency plus fallback-engagement latency plus the fallback's own settling time. If a pedestrian enters the path 1.2 s away and your monitor plus switch plus brake response consumes 0.9 s, you have 0.3 s of margin and no more; this budget, not detector accuracy alone, is what a reviewer will interrogate. Monitors must therefore run on a bounded-latency path, often a separate lightweight process or a safety microcontroller, never behind the same queue that the heavy network can stall.
Two further properties turn a set of monitors into an assurance argument. Coverage: for every hazard in the hazard analysis, some monitor must be able to observe its precursor; a hazard no monitor can see is an unmitigated gap you must be able to name. Soundness of the alarm-to-action map: every alarm must route to a response that is safe even if the alarm is a false positive, so the fallback is engineered to be benign-when-unnecessary (a smooth decelerate-and-alert), not disruptive. These two properties, plus monitor independence and the latency budget, are the evidence the safety case of Section 68.6 will cite. Agentic sensing systems that call tools and take actions (Chapter 22) need exactly this scaffolding, a checker gating every consequential action, before they can be trusted with anything irreversible.
Research Frontier
The open problem is monitoring perception in the semantic space where its failures actually live. Current SOTA moves beyond pixel-level OOD scores toward learned runtime monitors that watch the network's own activations: neuron-activation-pattern monitors (Cheng et al.'s work and the follow-on box-abstraction monitors) build an abstraction of "activation patterns seen on correct training predictions" and flag novel ones at inference, and the 2022 CVPR-era work on out-of-distribution detection for semantic segmentation pushes this into dense prediction. Newer directions synthesize monitors from Signal Temporal Logic over perception outputs, use conformal prediction to give the monitor a distribution-free false-alarm rate, and are formalized in ISO/PAS 8800 (2024) as the runtime-measures pillar of an AI safety argument. The hard, unsolved piece is a monitor with a provable detection guarantee against an adaptive adversary that has read the monitor.
Exercise
Extend the code above to a two-property monitor: keep the headway envelope, and add an STL "recovery" property, \(\square_{[0,T]}\!\bigl(\rho < 0 \Rightarrow \lozenge_{[0,k]}(\rho \ge m)\bigr)\), meaning "any violation must be recovered to margin \(m\) within \(k\) steps." Feed it a run where the fallback engages and re-establishes margin, and a run where it does not. Then measure: for a hazard arriving \(\Delta t\) seconds ahead, at what monitor latency does your reaction-time budget go negative? Plot the budget against detection latency and mark the point where the fallback can no longer save the scenario.
Self-Check
- Why does putting the OOD monitor on the same backbone as the perception model undermine the assurance argument, and what makes an independent monitor stronger?
- STL gives a robustness degree rather than a boolean. Name one concrete thing a runtime-assurance switch can do with the magnitude and gradient of robustness that it could not do with a bare pass/fail.
- A monitor achieves 99.9% detection accuracy but adds 200 ms of latency. Give a scenario in which it is useless despite that accuracy, and state the inequality it violates.
What's Next
In Section 68.5, we turn from building monitors to the standards that require them. ISO 26262, ISO 21448 (SOTIF), IEC 61508, and the new ISO/PAS 8800 for AI each define what "safe enough" means, how to allocate integrity levels, and where a runtime monitor sits in the evidence chain, giving the assurance machinery of this section its formal home.