Safety-critical constraints

"The furnace does not care about your validation accuracy. It cares whether the burner shuts off when the flame goes out."

A Sobered AI Agent

You come to this section assuming the probabilistic reasoning of Chapter 4 and the calibration machinery of Chapter 18. Those still matter, but a safety engineer will ask a different question: not "how accurate is your model on average?" but "what happens the moment it is wrong, and how do you prove it?" Averages are the wrong currency here. Safety is about bounding the worst case and demonstrating that bound to an auditor.

The independent protection layer principle

Process safety is built on layers of protection, each independent of the others, so that no single failure reaches a hazard. The model is drawn as concentric rings around the process: the basic process control system (the PLCs and loops of Section 35.2) sits closest; outside it sits alarms with operator response; outside that sits the safety instrumented system (SIS), a separate controller wired to separate sensors and final elements whose only job is to drive the process to a safe state; and outside everything sit mechanical relief devices such as rupture discs and pressure-relief valves that need no electronics at all. The what: a stack of diverse, independent barriers. The why: independence means the probability of a hazard is the product of each layer's failure probability, so five layers at \(10^{-1}\) each buy \(10^{-5}\) overall. The how: each layer uses different sensors, different logic solvers, and different physics wherever possible, because common-cause failure (one shared root defeating several layers) is what actually kills the multiplication.

A learned model belongs in the outer, advisory rings, not the innermost trip. It can sharpen an alarm, predict a trip before it happens, or recommend a setpoint, but the SIS that actually shuts the unit down remains a deterministic, certified, independently verifiable function. This is the same separation-of-concerns argument developed for autonomy in Chapter 68: the intelligent perception layer proposes, a simple safety monitor disposes.

SIL, SIS, and the language of functional safety

To work near these systems you must speak their vocabulary, codified in IEC 61508 (the general standard) and IEC 61511 (its process-industry sibling). A safety instrumented function (SIF) is one protective action, for example "close the fuel valve if furnace pressure exceeds the limit." Each SIF is assigned a safety integrity level (SIL 1 to SIL 4), a target for how reliably it must perform on demand. For a low-demand function the target is the average probability of failure on demand, \(\mathrm{PFD}_{avg}\), and the bands are logarithmic:

A SIL 3 function must fail to act on fewer than one demand in a thousand. You reach those numbers with redundant sensors, voting logic (a two-out-of-three arrangement tolerates one failed transmitter), regular proof testing, and hardware whose failure rates are themselves certified. The point for an AI engineer is scale: no current process for verifying a neural network yields a defensible \(\mathrm{PFD}_{avg}\) at SIL 2 or above, which is exactly why the learned component lives outside the SIF. Understanding the number tells you where the line is drawn and why.

Two more terms complete the working vocabulary. The fail-safe state is the condition the process moves to when anything goes wrong, chosen so that loss of power, loss of signal, or loss of air all drive toward safety (a spring-return valve that closes when de-energized is fail-safe). And fail-safe by de-energization means the safe action is the absence of a signal, so a cut wire or a crashed computer cannot prevent the trip. A model that must actively assert "everything is fine" to keep a machine running has the polarity backwards.

Latency, determinism, and the safety envelope

Beyond the layer architecture, safety-critical loops impose hard runtime constraints that a research-grade model rarely meets. Response time must be bounded, not merely fast on average: a burner trip specified at 200 ms means every trip, worst case, including garbage-collection pauses and cache misses. This rules out unbounded-latency inference in the trip path and pushes learned components onto the edge hardware of Chapter 59 with measured worst-case timing. Determinism matters too: the same inputs must produce the same action, which sits awkwardly with stochastic inference and with models that update online.

The practical pattern that reconciles a learned model with these constraints is the safety envelope (sometimes a runtime monitor or safety shield). The model proposes an action; a small deterministic function checks it against hard physical limits and either passes it through, clamps it, or overrides it with the fail-safe action. The guard is what you verify and certify. The listing below sketches this pattern: a learned valve-position advisor whose every command must survive a rule-based envelope before it can reach the actuator.

from dataclasses import dataclass

@dataclass
class SafetyEnvelope:
    p_max: float          # hard pressure limit (barg)
    pos_min: float        # min valve position (%)
    pos_max: float        # max valve position (%)
    watchdog_s: float     # max age of a fresh model command

    def arbitrate(self, model_cmd, pressure, cmd_age_s):
        """Deterministic guard: returns (action, reason). Fail-safe = close."""
        if cmd_age_s > self.watchdog_s:          # model stalled or crashed
            return 0.0, "watchdog_timeout_failsafe_close"
        if pressure >= self.p_max:               # hard trip, model ignored
            return 0.0, "overpressure_trip"
        clamped = min(max(model_cmd, self.pos_min), self.pos_max)
        if clamped != model_cmd:
            return clamped, "clamped_to_authorized_range"
        return clamped, "model_command_passed"

env = SafetyEnvelope(p_max=12.0, pos_min=5.0, pos_max=80.0, watchdog_s=1.0)
print(env.arbitrate(model_cmd=95.0, pressure=8.0, cmd_age_s=0.2))
print(env.arbitrate(model_cmd=40.0, pressure=12.5, cmd_age_s=0.2))
print(env.arbitrate(model_cmd=40.0, pressure=8.0, cmd_age_s=3.0))

Read the three printed cases against the taxonomy above: the first clamps an overconfident model to its authorized range, the second shows the hard pressure trip winning regardless of what the model wants, and the third shows fail-safe-on-silence when the model stops producing fresh commands. Nowhere does the model get the last word.

Verification, human oversight, and the audit trail

A safety case is not finished when the code runs; it is finished when you can convince an independent assessor. That demands three things a research workflow usually skips. First, traceability: every safety requirement links to the design element that implements it and the test that demonstrates it, so nothing is unaccounted for. Second, the model as a monitored input, not an unquestioned truth: the operator sees the model's recommendation and its confidence, retains the authority to reject it, and the interface is designed against automation complacency so that a plausible-looking wrong answer does not get rubber-stamped. Third, an immutable audit trail: what the model recommended, what the envelope did, and who acknowledged it, all timestamped, because after an incident the investigation will reconstruct exactly this sequence. These obligations connect directly to the regulatory and accountability discussion of Chapter 70 and, in the security dimension, to the cyber-physical threat model of Chapter 38, where an attacker who can spoof a sensor is trying to defeat exactly these layers.

The synthesis of this section is a division of labor. Let the learned model do what it is uniquely good at: fusing many noisy channels, spotting subtle precursors, and optimizing within a safe region. Let a small, boring, certified guard do what it is uniquely good at: guaranteeing that no matter how confidently wrong the model is, the process still ends up in a safe state. Confuse the two roles, and you have built something that demos beautifully and fails catastrophically. Keep them separate, and machine learning becomes a genuine asset on the plant floor rather than a liability the safety team has to fight.

What's Next

In Chapter 36, we move from protecting against the immediate hazard to forecasting the slow one: predictive maintenance and prognostics. Now that you know where a learned model may and may not sit in a safety architecture, we can let it do what it is best at, estimating remaining useful life from the degrading signals of a machine, so that intervention happens on your schedule rather than the failure's.