Safety cases for ML perception

"They asked whether my perception model was safe. I showed them 99.2 percent mAP. They asked again, more slowly, and I understood that a number is not an argument."

A Newly Humbled AI Agent

Claims, arguments, and evidence: the anatomy of a case

Every safety case has the same three-layer skeleton, made explicit by the Claims-Arguments-Evidence (CAE) framework and, more graphically, by Goal Structuring Notation (GSN). A goal (claim) is a proposition to be shown true. A strategy explains how you decompose that goal into sub-goals. Context and assumptions bound what the claim means. A solution is a pointer to concrete evidence (a test result, a proof, a field-data statistic). The argument is valid only if every leaf goal is discharged by a solution and every decomposition strategy actually entails its parent given the stated context.

Why bother with the formality? Because the failure mode of an informal safety argument is the hidden inductive leap: a step where "we tested a lot" silently becomes "therefore it is safe." Writing the argument as an explicit tree forces each leap into the open where a reviewer, or an adversarial safety assessor, can attack it. The Standardisation of Assurance-case Metamodel (SACM) from the Object Management Group gives this a machine-readable form, so a case can be version-controlled, queried, and diffed like code rather than living as a 400-page PDF nobody can audit.

The perception claim sits inside a wider hazard argument. Hazard analysis (from Section 68.5) yields a target such as "the residual rate of a hazardous missed detection shall be below \(\lambda_{\text{tol}}\)." The case must show the delivered system meets it. A useful way to keep the target quantitative is to decompose the top hazard rate multiplicatively across the layers you actually built:

where \(p_{\text{miss}}\) is the model's per-frame miss probability on the operational distribution, \(c_{\text{monitor}}\) the coverage of the runtime monitor that catches the miss, \(c_{\text{redundancy}}\) the independent-channel catch rate, and \(f_{\text{exposure}}\) the rate at which the hazardous scenario is encountered. Each factor becomes a sub-claim with its own evidence. The equation is the argument strategy made numeric: it shows exactly which measurement discharges which term, and it makes the reviewer's job checking that the numbers were co-measured on the same operational distribution rather than borrowed from three different reports.

The ML assurance gap and the AMLAS pattern

Classical assurance leans on a specification: the code is correct if it refines the spec, and coverage-directed testing exercises the spec's behaviours. Learned perception has no such spec. The "requirement" for pedestrian detection is a fuzzy human concept, the implementation is a weight tensor no human wrote, and the correctness proof would have to quantify over an input space of \(256^{H \times W \times 3}\) images. This is the assurance gap, and pretending it away is the single most common defect in early ML safety cases.

The AMLAS methodology (Assurance of Machine Learning for Autonomous Systems, from the University of York) and the emerging ISO/PAS 8800 both structure a case that confronts the gap directly. Rather than one claim about "the model," they decompose along the ML lifecycle, and require an argument at each stage that its output preserves the safety intent of the previous:

• Safety requirements allocation: the perceptual requirement is derived from and traces back to a system hazard, with performance and robustness targets that are verifiable, not aspirational.
• Data: the training and test data are relevant (drawn from the ODD), complete (cover the hazardous edge cases and rare classes), balanced, and accurate (labels are correct). A data argument is where most cases are weakest, because leakage or ODD gaps silently invalidate everything above.
• Model learning: the training process and the resulting model meet the allocated targets under the stated data assumptions.
• Model verification: independent evidence, held-out tests, formal robustness bounds, slice-based analysis, that the model satisfies its requirements, including on hard sub-populations.
• Deployment: the integrated model plus its runtime monitors and fallback behave safely, and the ODD is enforced at runtime so the model is never asked to operate where its evidence does not hold.

The subtle move is the confidence argument sitting beside the main one. For each leaf, you also state why the evidence is trustworthy: how much residual doubt remains, and why that doubt is tolerable. An "assurance deficit" that cannot be closed (for example, no test can cover every lighting condition) is not hidden; it is named, bounded, and mitigated by a downstream layer such as a monitor or a redundant channel. A case that lists its own deficits is stronger, not weaker, than one that claims none.

What counts as evidence for a perception claim

Evidence is not one kind of thing, and a strong case triangulates several so that no single weak method carries the argument. The main families, each mapped to the doubt it reduces:

• Statistical test evidence: held-out and slice-based performance with confidence intervals, and, crucially, a target error rate expressed with a statistical bound. To claim "miss rate below \(\lambda_{\text{tol}}\) with confidence \(1-\alpha\)" you need enough independent samples of the hazardous event; the rule of three says observing zero failures in \(n\) trials supports an upper bound near \(3/n\), which is why rare-hazard claims demand enormous, well-sampled test campaigns.
• Formal and robustness evidence: certified local robustness (a proof that no perturbation within an \(\ell_p\) ball flips the label) and reachability bounds. Strong where it applies, but local; it does not by itself cover semantic shift.
• Uncertainty and conformal evidence: calibrated confidence and conformal prediction sets (Chapter 18) give a distribution-free coverage guarantee that turns "the model is usually right" into "the true class is in the predicted set with probability \(\ge 1-\alpha\)," a claim a monitor can act on.
• Field and simulation evidence: proving-ground and shadow-mode fleet data, plus scenario coverage from a validated simulator or digital twin (Chapter 55). Simulation covers rare hazards testing cannot reach, at the cost of a sim-to-real validity sub-argument.
• Process and interpretability evidence: the development rigour (data governance, versioning) and root-cause traceability of failures (Chapter 67), which raise confidence that observed performance is not a fluke of the test set.

The discipline that ties these together is honest, single-pass measurement: the numbers that feed one argument branch must be co-computed on one model, one split, one operational replay, and saved as one artifact. A case that compares a miss rate from config A against a monitor coverage from config B has committed the exact audit-passes-but-invalid error the house evaluation rules warn against, and a competent assessor will find it.

from dataclasses import dataclass, field

@dataclass
class Node:
    id: str
    kind: str          # "goal", "strategy", or "solution"
    text: str
    children: list = field(default_factory=list)
    evidence: str = None   # only meaningful for solutions

def undischarged(node):
    """Return leaf goals with no supporting solution: the holes in the case."""
    holes = []
    if node.kind == "goal" and not node.children:
        holes.append(node.id)                       # a goal with nothing under it
    if node.kind == "solution" and not node.evidence:
        holes.append(node.id)                       # a solution pointing at no artifact
    for c in node.children:
        holes.extend(undischarged(c))
    return holes

case = Node("G1", "goal", "Miss rate in ODD < lambda_tol", children=[
    Node("S1", "strategy", "Argue over lifecycle stages", children=[
        Node("G2", "goal", "Data is relevant & complete", children=[
            Node("E1", "solution", "Leakage audit + slice coverage", evidence="rep_data.json")]),
        Node("G3", "goal", "Verification meets target", children=[
            Node("E2", "solution", "Per-slice held-out test", evidence="rep_eval.json")]),
        Node("G4", "goal", "Rain out-of-ODD is handled")   # deficit: no evidence yet
    ])])

print("Undischarged nodes:", undischarged(case))   # -> ['G4']

The snippet above shows why a machine-readable case matters: an argument you can traverse is an argument you can check. The undischarged function is the assurance analogue of a coverage report, it names the goals that reasoning has not yet reached, which is precisely where a rushed case papers over the gap.

Living safety cases: through-life assurance

A learned model's environment drifts, its data ages, and it may be updated after release, so a safety case frozen at launch decays. Two ideas keep it alive. First, the case is version-controlled and traceable: every claim links to the exact model hash, dataset version, and monitor configuration that discharges it, so when any of those changes you can query which claims are now stale. Second, the dynamic safety case treats runtime monitors (Section 68.4) as continuous evidence gatherers: field statistics on ODD coverage, monitor trigger rates, and drift indicators feed back to confirm, or invalidate, the assumptions the case rests on. When shadow-mode data shows the operational distribution moving away from the tested one (the shift detection of Chapter 66), the case flags the affected data and verification claims for re-argument before a hazard materialises. This closes the loop with the reliability-engineering patterns of the next section and with the fleet operations of Chapter 69, where re-training triggers and monitoring are the machinery that keeps the argument true after deployment.

What's Next

In Section 68.7, we turn the argument into engineering practice: the reliability-engineering patterns, FMEA and fault trees, graceful degradation, watchdogs, health management, and defense in depth, that make the sub-claims of a safety case actually true in silicon and field, and that generate the through-life evidence a living case feeds on.