Explainability and false-alarm triage

"The model was right about the arrhythmia and wrong about the loose electrode. The nurse could not tell which, so she stopped trusting both."

A Chastened AI Agent

Alarm fatigue is the real deployment failure

The dominant clinical failure mode of cardiac monitoring is not the missed beat; it is the flood of false ones. Studies of intensive-care telemetry have repeatedly found that the large majority of arrhythmia alarms are false or clinically irrelevant, driven by a lead coming loose, a patient brushing teeth, muscle tremor, or a single premature beat that meets a rule but means nothing. When most alarms are noise, clinicians habituate: response times lengthen, volumes get turned down, and alarms get pre-emptively disabled. This is alarm fatigue, and it is a documented cause of patient harm. The lesson for a model builder is that the operating-point metric that matters is rarely sensitivity alone. In a low-prevalence stream, positive predictive value dominates the human experience.

The arithmetic is unforgiving. Suppose a detector runs on a continuous stream where a genuine actionable event occurs at prevalence \(\pi = 0.001\) per analysis window. With sensitivity \(0.99\) and a specificity that sounds excellent at \(0.99\), the positive predictive value is

Nine in ten alerts are false at a specificity that would headline any paper. This is the same base-rate trap that governs any rare-event detector, and it is why the classical change-detection framing of Chapter 12 obsesses over false-alarm rate per hour rather than per-window specificity. The fix is not a better single classifier; it is a system that suppresses, defers, and explains.

Explaining an ECG decision: attribution that lands on P, QRS, and T

Explainability for ECG has a rare advantage over most deep-learning interpretability: the ground-truth vocabulary already exists. A cardiologist reasons in terms of named landmarks (a widened QRS, a flattened P, ST-segment elevation, a prolonged QT), so a faithful explanation should point at those same regions of the trace. This turns a vague "which pixels mattered" question into a checkable one: does the model's evidence overlap the physiological feature a clinician would cite?

The workhorse tools are gradient-based attribution over the time axis. Grad-CAM adapted to 1D convolutions produces a per-sample importance curve you can overlay on the waveform; Integrated Gradients attributes the prediction to input samples relative to a baseline (a flat or median-beat reference) with the appealing completeness property that attributions sum to the score difference. For the transformer and state-space ECG models of Section 29.5, attention weights and attention-rollout give a complementary view of which beats and which intervals the model pooled over. The key discipline, developed generally in Chapter 67, is to not trust a saliency map because it looks plausible. Validate it: perturb the QRS and confirm the attribution and the prediction both move; run a sanity check where randomizing the top layer's weights should destroy the map. A saliency method that survives neither test is decoration.

The highest-value explanation is often not a heatmap but a concept: report that the model's decision is driven by "QRS duration above threshold" or "irregular RR intervals," phrased in the clinician's ontology. Prototype and concept-based models learn this directly, matching a query beat against learned exemplars ("this looks like these three training beats labeled PVC"), which a human can inspect far faster than an abstract gradient field.

Triage as a pipeline: quality gates and confidence cascades

False-alarm triage is best built as a cascade of cheap-to-expensive stages, each empowered to suppress or defer rather than to alarm. The first and most cost-effective stage is a signal-quality index (SQI). Before any classifier runs, ask whether the window even contains an interpretable ECG. Cheap SQI features include the fraction of the window that is flat or railed, the power ratio between the QRS band (roughly 5 to 40 Hz) and out-of-band noise, kurtosis, and the agreement between two independent QRS detectors on the same strip. A window that fails SQI is not classified as normal or abnormal; it is labeled "uninterpretable" and either suppressed or routed for re-acquisition. Because most artifact-driven false alarms originate in low-quality windows, this single gate removes a large share of them before the model is even consulted, and it is exactly the kind of gating the wearable pipelines of Chapter 30 apply to motion-corrupted PPG.

The second stage is a confidence and uncertainty gate. A well-calibrated probability (Chapter 18) lets you route the model's own low-confidence outputs to a human or to a second opinion instead of alarming automatically. Conformal prediction gives this a distribution-free guarantee: rather than emit a point label, emit a prediction set, and treat a window whose set contains more than one arrhythmia class as "ambiguous, escalate." A third stage can add temporal persistence, requiring an abnormal call to hold across several consecutive windows before it becomes an alarm, which trades a few seconds of latency for a large drop in one-off false triggers. The design question at each stage is the cost asymmetry: a suppressed true ventricular tachycardia is far more expensive than a suppressed premature beat, so persistence and confidence thresholds should be class-specific, not global.

import numpy as np

def signal_quality_index(x, fs):
    """Cheap SQI in [0,1]: high when the window looks like a clean ECG."""
    # 1) fraction of samples that are flat (lead-off / saturation)
    flat = np.mean(np.abs(np.diff(x)) < 1e-4)
    # 2) in-band (QRS, 5-40 Hz) vs total power via a crude spectral ratio
    X = np.abs(np.fft.rfft(x - x.mean()))**2
    f = np.fft.rfftfreq(len(x), 1/fs)
    inband = X[(f >= 5) & (f <= 40)].sum()
    ratio = inband / (X.sum() + 1e-12)
    return float(np.clip(ratio * (1 - flat), 0, 1))

def triage(x, fs, model, sqi_min=0.5, p_high=0.9, p_low=0.6):
    """Return an action, not just a label, for one analysis window."""
    q = signal_quality_index(x, fs)
    if q < sqi_min:
        return "SUPPRESS_UNINTERPRETABLE", q     # bad signal: re-acquire, do not alarm
    p = model.predict_proba(x)                   # calibrated abnormal-class prob.
    if p >= p_high:
        return "ALARM", p                        # confident and clean -> alert human
    if p >= p_low:
        return "ESCALATE_REVIEW", p              # ambiguous -> queue for a second look
    return "CLEAR", p                            # confident normal -> stay silent

# Every abnormal-looking window is now sorted into alarm / review / suppress,
# instead of every model positive becoming an alarm.

The code above is deliberately spartan so the control flow is visible. Notice that the model is only one of four decisions: the SQI gate, the two confidence thresholds, and (omitted here) temporal persistence do most of the alarm reduction. This is the recurring shape of a trustworthy monitor: the classifier proposes, and the surrounding logic disposes.

From explanation to threshold: cost-sensitive operating points

Explainability and triage meet at the operating point. Once explanations have let you audit why the model fires, you set thresholds by explicit cost rather than by a default \(0.5\). Let \(C_{\text{FN}}\) be the cost of a missed event and \(C_{\text{FP}}\) the cost of a false alarm (including its contribution to fatigue). The Bayes-optimal threshold on a calibrated probability is

so a class where a miss is catastrophic (ventricular tachycardia) gets a low threshold and fires readily, while a benign-but-noisy class (isolated ectopy) gets a high one. Because these thresholds only make sense on a calibrated score, the calibration work of Chapter 18 is not optional polish; it is what makes the cost model coherent. The final honesty check is evaluation design: alarm-reduction numbers must be measured at fixed sensitivity, patient-by-patient rather than beat-by-beat, so a system cannot look good by suppressing alarms on the easy patients while missing events on the hard ones. That patient-level, leakage-safe framing is the subject of the next section.

What's Next

In Section 29.7, we turn the triage and calibration choices of this section into a defensible study. We cover clinical validation design: patient-level and site-level splits that respect the leakage traps, prospective versus retrospective evaluation, the reference-standard and adjudication problem, subgroup reporting, and the regulatory framing that decides whether a cardiac model is a research demo or a device a clinician is allowed to rely on.