"Retrospective accuracy is a promise. Prospective validation is the part where the universe checks whether you meant it."
An Empirically Humbled AI Agent
Prerequisites
This section assumes the regulatory framing of Section 34.1 (what makes a model Software as a Medical Device and what the regulator expects it to demonstrate). It leans on the base-rate and confidence-interval reasoning of Chapter 4, the patient-level, leakage-safe splitting discipline of Chapter 5, and the evaluation-protocol vocabulary of Chapter 65. Change control for a model that keeps learning after the trial is the topic of Section 34.2; here the model is frozen for the duration of the study.
The Big Picture
Every number in this book so far came from data you already held: a benchmark, a held-out split, a retrospective cohort. A clinical claim cannot be made that way. To say "this wearable detects the condition in real patients" you must freeze the model, write down what you expect it to do before you see the answer, then run it forward on patients you have not touched, against a reference standard you trust, and compare. That forward run is prospective validation, and it is a different epistemic act from retrospective evaluation. Retrospective analysis is where you form the hypothesis; the prospective trial is where you risk being wrong in public. This section is about designing that trial: choosing the endpoint, the population, the reference standard, and the sample size so that the result means what a regulator, a clinician, and a patient need it to mean.
Why retrospective accuracy is not clinical evidence
A model that scores beautifully on a curated retrospective dataset has cleared the lowest bar, not the highest. The retrospective set was assembled with hindsight: someone chose which records to include, which to label, and often which "difficult" or "ungradable" cases to quietly drop. Thresholds were tuned until the curve looked good. Even with scrupulous patient-level splitting, the whole dataset shares one collection era, a handful of sites, and a labeling pipeline the model may have implicitly co-evolved with. None of that survives contact with a new hospital, a new device firmware, or next winter's patient mix. Prospective validation removes the two degrees of freedom that inflate retrospective numbers: the model is locked before enrollment, and the patients are whoever walks in, in the order they walk in. You are no longer selecting the exam; you are sitting it.
Why insist on pre-registration? Because a study you can reinterpret after seeing the data proves nothing. Writing the primary endpoint, the success threshold, the analysis population, and the statistical test into a public protocol (a registry entry, an analysis plan agreed with the regulator) before the first patient enrolls is what converts a demonstration into evidence. It stops the most common form of self-deception: running the trial, finding the pre-specified endpoint missed, and discovering a secondary subgroup where the model "actually" worked.
Key Insight
There are three claims, and they are not interchangeable. Analytic validation asks: does the algorithm compute its output correctly and reproducibly from the input signal? Clinical validation asks: does that output correspond to the clinical condition in the intended-use population? Clinical utility asks: does acting on the output make patients better off? A model can pass the first, pass the second, and still fail the third, because a correct, accurate prediction that no one can act on, or that triggers harmful downstream care, changes no outcome. Most wearable AI ships having proven analytic and clinical validity and merely asserting utility. Know which claim your trial is powered to support, and do not let the marketing team upgrade it.
The anatomy of a prospective validation study
A defensible study fixes six things before enrollment. The intended-use population defines who is eligible, in enough detail that spectrum bias cannot creep in: a screener validated only on symptomatic patients referred to a specialist will underperform when pointed at an asymptomatic general population, because the easy-hard mix has shifted. The reference standard is the source of truth the model is graded against, and it must be independent of the model and at least as good as the clinical decision it informs (an adjudicating panel, a gold-standard instrument, a follow-up outcome). The primary endpoint is the single pre-specified metric that defines success, with its acceptance threshold; secondary endpoints are supporting evidence, never a fallback. Blinding keeps the people applying the reference standard from seeing the model's output, so the truth is not contaminated by the thing being tested. The enrollment and site plan ensures the patients and the operating conditions match deployment: if community nurses will run the device, the trial operators should be community nurses, not the research team that built it. And the analysis plan names the statistical test, the confidence-interval method, and the handling of missing or ungradable cases.
Reporting guidelines exist precisely so none of this is improvised. STARD governs diagnostic-accuracy studies; SPIRIT-AI and CONSORT-AI extend clinical-trial protocol and reporting standards to AI interventions; and DECIDE-AI covers the early live-deployment evaluation of decision-support systems. Adopting the relevant checklist at design time, not at write-up time, is the cheapest quality gate you will ever install. It also aligns the trial with the leakage-safe evaluation discipline that runs through Chapter 65: the reference standard, the analysis population, and the operating point are all fixed in advance and reported in full.
In Practice: the autonomous diabetic-retinopathy trial
The first FDA-authorized fully autonomous diagnostic AI, IDx-DR (now marketed as LumineticsCore), earned that status through a pre-registered prospective study reported in npj Digital Medicine in 2018. Roughly 900 patients with diabetes were enrolled across ten primary-care sites, and the fundus images were captured by existing clinic staff, not ophthalmologists, exactly the intended-use operators. The system's output was compared against a reference standard built from a fundus-photograph reading center plus optical coherence tomography, adjudicated independently of the algorithm. Crucially, the success thresholds were pre-specified with the regulator (superiority against 85 percent sensitivity and 82.5 percent specificity), and the trial reported sensitivity around 87 percent and specificity around 91 percent, clearing the pre-set bar. The design lessons generalize to any wearable claim: freeze the model, use real-world operators, grade against an independent truth, and write the winning line before you run the race. A retrospective re-read of the same images by the same builders would have proven none of it.
Powering the study: how many patients, and how many sick ones
The single most common design error in wearable-AI trials is enrolling for total headcount instead of for diseased cases. A diagnostic study's precision on sensitivity is governed by the number of true-positive-eligible (diseased) patients, and its precision on specificity by the number of non-diseased patients, not by the grand total. Because the target condition is usually rare, the diseased count is the binding constraint, and prevalence sets how many people you must screen to reach it. For a target sensitivity \(Se\) that you want to estimate to within a confidence-interval half-width \(W\) at confidence level \(1-\alpha\), the required number of diseased patients follows the standard proportion formula
$$ n_{\text{pos}} \;=\; \frac{z_{1-\alpha/2}^{2}\; Se\,(1-Se)}{W^{2}}, \qquad N_{\text{total}} \;=\; \frac{n_{\text{pos}}}{\pi}, $$where \(\pi\) is the prevalence of the condition in the enrolled population and \(N_{\text{total}}\) is the resulting total enrollment. The listing below turns these two formulas into a planner and shows why a low-prevalence screening claim is expensive: the diseased target is modest, but the enrollment it implies is not.
import numpy as np
from scipy.stats import norm
def plan_sample_size(target_metric, half_width, prevalence, alpha=0.05):
"""Diseased cases and total enrollment for a diagnostic-accuracy study."""
z = norm.ppf(1 - alpha / 2)
n_cases = (z ** 2) * target_metric * (1 - target_metric) / (half_width ** 2)
n_cases = int(np.ceil(n_cases))
n_total = int(np.ceil(n_cases / prevalence))
return n_cases, n_total
for prev in [0.30, 0.05, 0.01]:
cases, total = plan_sample_size(target_metric=0.90, half_width=0.05, prevalence=prev)
print(f"prevalence={prev:>5.0%} diseased needed={cases:4d} enroll={total:6d}")As Listing 34.3 makes concrete, the same 90 percent sensitivity claim costs about 460 enrollees in an enriched, high-prevalence cohort and roughly thirty times that in a general screening population. This is why enrichment (enrolling a population where the condition is more common) and reference-standard cost dominate trial planning. It is also why the base-rate arithmetic of Chapter 4 belongs in the design phase, not the post-hoc discussion: the prevalence you will actually deploy into decides whether a strong sensitivity even yields an actionable positive predictive value.
The Right Tool
The formula above is fine for a headline estimate, but a real protocol needs power for a hypothesis test (superiority against a pre-specified threshold), exact rather than Wald intervals near the boundaries, and adjustment for clustering when one patient contributes many windows. Rolling that yourself is a few hundred lines of statistics you can get subtly wrong. statsmodels supplies the power machinery directly:
from statsmodels.stats.proportion import proportion_confint, samplesize_confint_proportion
from statsmodels.stats.power import NormalIndPower
# exact-ish CI for an observed sensitivity, and n for a target CI width
lo, hi = proportion_confint(count=126, nobs=139, method="wilson")
n_cases = samplesize_confint_proportion(proportion=0.90, half_length=0.05)statsmodels replaces roughly 200 lines of interval and power code with library calls that handle Wilson and exact intervals plus one- and two-sided power. It computes the statistics; specifying the intended-use population, the reference standard, and the clustering structure remains a design decision the library cannot make for you.The tool removes the arithmetic, not the judgment. A power calculation that ignores within-patient correlation, the same leakage failure the patient-level splitting of Chapter 5 guards against, will report far more effective samples than you truly have and leave the trial underpowered.
Silent trials before the model touches care
Between the retrospective lab and the pre-registered pivotal trial sits a cheap, powerful intermediate step: the silent trial, also called a shadow deployment. The frozen model runs live on real incoming patient data and records what it would have alerted, but its output is hidden from clinicians and changes no care. A silent period surfaces the failures that no retrospective set predicts: firmware drift, a new patient demographic, sensor placement the training data never saw, seasonal shifts in the incoming mix. It lets you measure the real-world alert rate and calibration before a single decision hangs on the model, and it gives the monitoring infrastructure of Chapter 69 a rehearsal. Only after a silent trial confirms the operating point holds in the wild should the model graduate to an interventional trial where its output reaches a clinician. Throughout, the probabilities it emits must be calibrated, not merely discriminative, which is why the conformal and calibration methods of Chapter 18 are a prerequisite for any trustworthy alert threshold.
Exercise
You are designing a prospective study for a wrist-wearable that flags a condition with 2 percent prevalence in the screened population. Your regulator wants sensitivity estimated to within plus-or-minus 4 points at 95 percent confidence, assuming a true sensitivity near 0.88. Using the formulas in Listing 34.3, compute the number of confirmed cases required and the total enrollment. Then state, in one sentence, one design change that would cut the enrollment without weakening the sensitivity claim, and name its cost.
Self-Check
1. Distinguish analytic validation, clinical validation, and clinical utility. Which one does a diagnostic-accuracy study typically leave unproven, and why?
2. Why does the number of diseased patients, rather than total enrollment, set the precision of a sensitivity estimate, and what role does prevalence play?
3. What does a silent trial reveal that a retrospective held-out set cannot, and why must it precede any interventional trial?
What's Next
In Section 34.4, we turn from proving that a physiological model works to the uncomfortable fact that the very signals it consumes are themselves identifying: an ECG, a gait trace, or a photoplethysmogram can act as a biometric, and we examine how much a supposedly de-identified sensor stream can leak about who its owner is.