Documentation and accountability

"When the incident review asked why I flagged that pump as healthy, I could produce the exact model version, the sixty seconds of vibration I saw, and the confidence I reported. What nobody had written down was who decided my ninety percent was good enough to skip an inspection."

A Thoroughly Logged AI Agent

The accountability chain: from an output back to a person

Every consequential decision a sensor system makes should trace back along an unbroken chain to a human who is answerable for it. Break the chain at any link and accountability evaporates. Consider a wearable that decides not to raise a fall alert. To be accountable, that non-decision must connect to the model version that made it, the model to the training data and evaluation that qualified it, the evaluation to a documented operating envelope (the conditions under which the numbers hold), the envelope to a deployment threshold, and the threshold to a named owner who signed off that this threshold was acceptable for this population. Documentation is simply the material that makes each link inspectable. The failure mode you are guarding against is not usually a missing model; it is a missing reason. "The model said so" is not a reason a court, a hospital safety board, or a grieving family will accept, and it is not one you should accept from your own system either.

The important consequence is that documentation must be produced at the moment of the decision, not reconstructed afterward. Reconstruction is where accountability quietly dies: the model was retrained last week, the config drifted, the raw signal was already discarded for privacy, and the person who set the threshold left the company. An accountable pipeline captures the who, what, and why as immutable records synchronized to the decision itself, so that six months later the answer exists whether or not anyone remembers the incident.

Model cards and system cards for sensor AI

A model card is a short, structured document that states what a model is for, how well it works, and where it must not be used. The concept comes from Mitchell and colleagues, and the sensor setting adds fields that a generic vision or language card omits. A sensor model card must declare its modality and hardware assumptions (a fall-detector trained on a wrist IMU at \(50\,\text{Hz}\) is undefined on a hip-worn unit at \(25\,\text{Hz}\)), its operating envelope (sampling rate, orientation, mounting, temperature, motion regime), its subgroup performance across bodies and environments as required by Section 70.3, its calibration and confidence semantics so a downstream consumer knows what a reported \(0.9\) means, and its explicit out-of-scope uses. A system card zooms out one level to document how models, thresholds, human oversight, and fallback behavior compose into the deployed product; it is the artifact a functional-safety review (Chapter 68) reads first.

The discipline that makes cards worth the effort is that the numbers on them are co-computed in one pass on one evaluation configuration and cite the leakage-safe split they came from. A card that reports overall accuracy from one run and per-subgroup accuracy from a different, more favorable run is worse than no card: it launders an invalid comparison behind an official-looking template. Treat the card as a signed claim, not a marketing surface.

Audit trails: replaying any decision

An audit trail is the append-only record that lets you reconstruct a specific past decision. For sensor AI the minimum unit is a decision record that binds five things: a timestamp, a hash of the model version and configuration, a hash or reference to the input window (not necessarily the raw signal, which privacy may forbid you to keep), the output and its calibrated confidence, and the identity of the policy or human that acted on it. Hashing rather than storing raw sensor data is what lets the trail coexist with the data-minimization duty of Section 70.1: you can prove which input was seen without retaining a re-identifiable biometric trace. The code below constructs such a record and shows why the hash matters: change the model or the window by one byte and the recomputed digest no longer matches, so tampering and silent version drift both become detectable.

import hashlib, json, time
from dataclasses import dataclass, asdict

def digest(obj) -> str:
    """Stable content hash of any JSON-serializable object."""
    blob = json.dumps(obj, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()[:16]

@dataclass(frozen=True)
class DecisionRecord:
    ts: float
    model_id: str            # e.g. "fall-detect-v3.2"
    model_hash: str          # digest of weights + config
    input_hash: str          # digest of the input window (raw signal not stored)
    output: str
    confidence: float
    acted_by: str            # policy id or human sign-off owner

def log_decision(model_id, model_cfg, window, output, confidence, acted_by):
    rec = DecisionRecord(
        ts=time.time(),
        model_id=model_id,
        model_hash=digest(model_cfg),
        input_hash=digest(window),      # window = list of sensor samples
        output=output,
        confidence=round(confidence, 4),
        acted_by=acted_by,
    )
    return asdict(rec)

rec = log_decision("fall-detect-v3.2", {"arch": "tcn", "fs": 50, "thr": 0.62},
                   window=[0.98, 1.01, 0.15, 2.4], output="no_alert",
                   confidence=0.91, acted_by="policy:auto-suppress-below-0.62")
print(rec)

Incident records and the corrective loop

Accountability is proven not when nothing goes wrong but when something does. A post-incident record documents a failure and closes a loop: what the system did, what it should have done, which link in the chain broke, and what change prevents a recurrence. The value is in making failures legible and cumulative rather than embarrassing and forgotten. A disciplined incident record names a root cause at the level where the fix lives (a mislabeled training segment, a threshold set for a population the deployment did not match, an uncalibrated confidence), not a scapegoat, and it links to the corrective action and the model version that carries the fix. Over a fleet, these records become the institutional memory that keeps the same failure from shipping twice.

Keeping documentation alive

The predictable way all of this fails is drift: the card describes v2 while the fleet runs v4, the audit schema stops matching the pipeline, and the incident log fills with prose nobody queries. Static documents written once and filed are worse than useless because they radiate false confidence. The fix is to treat documentation as a build artifact, regenerated in the same CI job that trains and evaluates the model, versioned in the same repository, and gated so a deploy is blocked if the card's metrics were not co-computed on the current evaluation run. In regulated settings such as medical devices, this living dossier is close to what Chapter 34 calls the technical file; in the ordinary case it is simply the difference between documentation you can trust and paperwork you cannot. When the artifact is generated from the run, it cannot lie about the run, and that single property is what turns documentation from a liability into the backbone of an accountable system.

What's Next

In Chapter 71, we leave the cross-cutting principles behind and assemble everything into domain playbooks, where the documentation, privacy, and accountability duties of this chapter become concrete checklists for wearables, robotics, vehicles, smart spaces, environmental monitoring, and security, each with its own regulator, failure modes, and evidence a deployer must be ready to produce.