Clinical validation design

"My AUC was 0.94 on the held-out set. Then a real clinic ran me at a real prevalence, and I learned that nine out of ten alarms I raised were wrong. The test set never told me that."

A Chastened AI Agent

This section assumes the leakage-safe splitting discipline of Chapter 5 and the general evaluation-protocol machinery of Chapter 65, and it leans on the calibration and uncertainty ideas from Chapter 18. It is the cardiac-specific complement to Chapter 34, which covers the regulatory pathway and biometric-privacy law in full; here we stay on study design. We do not re-cover the false-alarm triage of Section 29.6 or the models of Section 29.5; we assume a model exists and ask how to prove it is worth deploying.

The validation ladder: from discrimination to outcomes

Clinical validation is not one study but a ladder, and each rung answers a strictly harder question than the one below it. Rung one, retrospective internal validation, holds out records from the development database and reports discrimination (AUROC, sensitivity, specificity). It is cheap, fast, and almost worthless as evidence of real-world value, because the held-out records share the same recruitment, hardware, and labeling pipeline as training. Rung two, external validation, runs the frozen model on data from institutions, devices, and populations that never touched development. This is where most impressive numbers deflate: an ECG model trained in one health system routinely loses several points of AUROC and, worse, loses calibration when it meets a new patient mix, a topic treated generally under distribution shift in Chapter 66. Rung three, prospective silent evaluation, deploys the model live but hides its output from clinicians, so you measure performance on the true incoming stream without letting the model affect care. Rung four, the pragmatic randomized trial, finally shows the output to clinicians in one arm and not the other, and measures a clinical outcome. Only rung four proves the model changes what happens to patients. The mistake that sinks products is stopping at rung one and reporting it as if it were rung four.

Metrics that survive deployment: prevalence, PPV, and net benefit

The single most common failure in cardiac-AI validation is reporting sensitivity and specificity, which are properties of the model, and forgetting positive predictive value, which is a property of the model and the population. By Bayes' rule, at prevalence \(\pi\), sensitivity \(\text{Se}\), and specificity \(\text{Sp}\),

For a low-ejection-fraction screen with \(\text{Se}=0.90\), \(\text{Sp}=0.90\), the PPV is a healthy \(0.89\) at \(\pi=0.5\) but collapses to \(0.08\) at a screening prevalence of \(\pi=0.01\). Nine of ten positive alarms are false, and every one triggers a downstream echocardiogram. This is the number a hospital committee cares about, and it is why validation must be run and reported at the deployment prevalence, not the enriched case-control prevalence of the development set. Beyond raw predictive values, decision-curve analysis (Vickers and Elkin, 2006) plots net benefit against the threshold probability a clinician would accept, folding the relative cost of a missed case versus a false alarm into a single curve; it answers "is acting on this model better than treat-all or treat-none" across the plausible range of clinical preferences. Report calibration too: a model can rank perfectly (high AUROC) yet systematically over-predict risk, and a miscalibrated probability is dangerous the moment a clinician reads it as a real chance.

import numpy as np

def ppv_npv(sens, spec, prevalence):
    """Predictive values at a target deployment prevalence."""
    tp = prevalence * sens
    fn = prevalence * (1 - sens)
    tn = (1 - prevalence) * spec
    fp = (1 - prevalence) * (1 - spec)
    return tp / (tp + fp), tn / (tn + fn)

def net_benefit(sens, spec, prevalence, threshold):
    """Decision-curve net benefit of the model at one threshold prob."""
    tp = prevalence * sens
    fp = (1 - prevalence) * (1 - spec)
    w = threshold / (1 - threshold)          # odds of the decision threshold
    return tp - fp * w                       # per-patient net benefit

for pi in (0.50, 0.05, 0.01):
    ppv, npv = ppv_npv(0.90, 0.90, pi)
    nb = net_benefit(0.90, 0.90, pi, threshold=0.05)
    print(f"prev={pi:>4}: PPV={ppv:0.2f}  NPV={npv:0.3f}  net_benefit={nb:+0.4f}")

Splits that mirror the real generalization gap

A validation split is a hypothesis about how the model will be challenged in the world, and the split must match the challenge. Patient-level splitting is non-negotiable: multiple ECGs from one patient sharing training and test sets leaks identity and inflates every metric, the same beat-versus-record leakage warned about in Section 29.4. Beyond that floor, three stratified splits probe three real shifts. A temporal split (train on 2018 to 2021, test on 2023) exposes drift in acquisition hardware, coding practice, and patient case-mix over time. A site or geographic split (train on hospitals A and B, test on hospital C) exposes the institution-specific confounds that make single-site AUROC a fiction. A subgroup-stratified report breaks performance down by age, sex, and device type, because an aggregate number can hide that the model fails on exactly the group the confound was correlated with; fairness auditing across such subgroups returns in force for wearables in Chapter 30. The prospective validation cohort should be defined the way a clinical trial defines its population: with pre-registered inclusion and exclusion criteria and a locked analysis plan, so you cannot fish for the split that flatters the model.

Reader studies: standalone, assistive, and the human in the loop

A cardiac model is rarely deployed alone; it is deployed alongside a clinician, so the validation must measure the human-model team, not just the model. Three designs answer different questions. A standalone study compares the model against a reference standard (an adjudicated panel, an echocardiogram, a serum result) and reports how the algorithm does by itself. A reader study compares unaided clinicians against the model on the same cases. The design that actually predicts deployment is the multi-reader, multi-case (MRMC) assistive study: the same readers interpret the same cases twice, once unaided and once with the model's output, and the analysis isolates the incremental effect of the assistance while accounting for both reader and case variability. MRMC is the design regulators expect for assistive tools precisely because it answers the deployment question: does showing the output make clinicians better, worse (via automation bias or alarm fatigue), or unchanged? An assistive tool can have a superb standalone AUROC and still harm care if clinicians defer to its false positives, which is why the human-in-the-loop measurement, not the standalone number, is the one that governs the label.

What's Next

In Chapter 30, we leave the twelve-lead clinical ECG for the photoplethysmogram on your wrist, where the validation problems of this section get harder: prevalence is lower, signal quality is worse, and the population screened is everyone with a watch, which is where PPV, subgroup fairness, and prospective design stop being academic and start deciding whether a consumer alert helps or terrifies.