"An average that works for everyone on average is a device that works for no one in particular."
A Subgroup-Aware AI Agent
Prerequisites
This section builds on the sensor-physics intuition of Chapter 2 (why the transduction path itself can be biased before any model touches the signal), the optical cardiovascular sensing of Chapter 30, the leakage-safe, patient-level dataset discipline of Chapter 5, and the calibration and confidence-interval reasoning of Chapter 18 and Chapter 4. Fairness is measured on the same held-out, pre-registered footing as the prospective validation of Section 34.3; here we simply refuse to report a single pooled number.
The Big Picture
A health sensor that is accurate on average can still be dangerous, because "on average" hides who it fails. When a wearable pulse oximeter systematically overestimates blood oxygen on darker skin, the pooled sensitivity looks fine while a specific group of patients is quietly denied care they qualify for. Fairness in health sensing is not a compliance checkbox bolted on after training; it is a property of the whole pipeline, from the physics of the transducer, through the demographics of the training set, to the operating threshold you ship. This section is about making the failure visible: slicing performance by the groups who bear the risk, choosing the right parity criterion for the clinical decision, and deciding what to do when the sensor itself, not the model, is the source of the gap.
Where the bias enters: physics, sampling, and labels
Unfairness in a health model rarely originates in the loss function. It enters upstream, in three places that a fairness audit must separate because they demand different fixes. The first is measurement bias in the transducer: the sensor physics itself responds differently across bodies. Photoplethysmography (the green- and red-light optical method behind wrist heart-rate and pulse oximetry, see Chapter 30) depends on light penetrating tissue and scattering back off pulsating blood. Melanin absorbs strongly in exactly the wavelengths these sensors use, so the raw signal-to-noise ratio is lower on darker skin, and a calibration curve fit mostly on lighter skin will read biased. No amount of clever modeling fully repairs a signal the hardware never captured well; this is a Chapter 2 problem wearing a machine-learning costume.
The second is sampling bias in the training data: if the cohort that generated your labels skews young, light-skinned, male, or drawn from one hospital, the model minimizes error where the mass of data sits and treats sparse subgroups as noise. The third is label bias: the reference standard itself can encode inequity, for example if a diagnosis was historically under-recorded in a group with less access to care, so the "ground truth" the model learns to imitate is already skewed. A parity gap you find in the audit could come from any of the three, and the mitigation differs: recalibrate the sensor, re-weight or re-collect data, or repair the labels. Diagnosing which one you have is the real work.
Key Insight
There is no single fairness metric, and the choice is a clinical decision, not a statistical one. Performance parity asks that sensitivity and specificity match across groups: appropriate when a missed detection is the harm, as in a screening alert. Calibration parity asks that a predicted risk of 0.2 means a true rate of 0.2 in every group: appropriate when clinicians act on the probability itself. Equalized odds constrains both error rates jointly. These criteria can be mathematically incompatible: when base rates differ across groups, you generally cannot have equal calibration and equal false-positive rates at once. So you must name the decision the output feeds, pick the criterion that bounds the actual harm, and state the tradeoff you accepted, rather than optimizing a fairness number you never justified.
Auditing: slice, count, and put an interval on every group
The mechanics of a fairness audit are disciplined disaggregation. You take the same frozen model and the same pre-registered test set from Section 34.3, then report the primary metric separately for each protected subgroup, each with a confidence interval, never as a pooled scalar. Two traps recur. The first is silent underpowering: a subgroup with forty patients can show a sensitivity of 0.80 with a confidence interval so wide it overlaps both 0.6 and 0.95, meaning you have measured almost nothing about that group. A fairness table without per-group sample sizes and intervals is theater. The second is intersectionality: a model can be fair on skin tone marginally and fair on sex marginally while failing badly for darker-skinned women specifically, because the harm concentrates in a cell that neither one-dimensional slice reveals. The audit should include the intersecting cells that are large enough to estimate, and flag the ones too small to trust as an explicit gap in evidence, not a pass.
The listing below performs the core of such an audit: it slices predictions by group, computes sensitivity and specificity with Wilson intervals (which behave near the 0 and 1 boundaries where subgroup counts are small), and reports the worst-group gap that a reviewer should have to explain.
import numpy as np
from statsmodels.stats.proportion import proportion_confint
def fairness_audit(y_true, y_pred, group, positive=1):
"""Per-group sensitivity/specificity with Wilson CIs and the worst gap."""
rows = {}
for g in np.unique(group):
m = group == g
pos, neg = (y_true[m] == positive), (y_true[m] != positive)
tp = int(((y_pred[m] == positive) & pos).sum())
tn = int(((y_pred[m] != positive) & neg).sum())
se_lo, se_hi = proportion_confint(tp, max(pos.sum(), 1), method="wilson")
sp_lo, sp_hi = proportion_confint(tn, max(neg.sum(), 1), method="wilson")
rows[g] = dict(n=int(m.sum()), n_pos=int(pos.sum()),
sens=tp / max(pos.sum(), 1), sens_ci=(se_lo, se_hi),
spec=tn / max(neg.sum(), 1), spec_ci=(sp_lo, sp_hi))
sens_vals = [r["sens"] for r in rows.values()]
return rows, max(sens_vals) - min(sens_vals) # sensitivity disparity
# toy example: group B is both under-sampled and under-served
rng = np.random.default_rng(0)
grp = np.array(["A"] * 400 + ["B"] * 60)
y = rng.integers(0, 2, size=460)
p = y.copy()
flip = rng.random(460) < np.where(grp == "B", 0.30, 0.08) # more errors in B
p[flip] = 1 - p[flip]
rows, gap = fairness_audit(y, p, grp)
for g, r in rows.items():
print(f"{g}: n={r['n']:3d} pos={r['n_pos']:3d} "
f"sens={r['sens']:.2f} CI[{r['sens_ci'][0]:.2f},{r['sens_ci'][1]:.2f}]")
print(f"sensitivity disparity = {gap:.2f}")As Listing 34.6 makes concrete, the disparity number is only meaningful alongside the per-group interval widths. A 0.15 sensitivity gap where both intervals are tight is a real, shippable-only-with-a-fix problem; the same gap where the small group's interval spans 0.3 is a data-collection problem masquerading as a fairness problem. The audit is run on the leakage-safe, patient-level split of Chapter 5: if windows from one patient straddle train and test, per-group accuracy is inflated exactly where you most need it honest.
In Practice: the pulse-oximeter oxygen gap
During the 2020 COVID-19 surge, clinicians and researchers documented that finger pulse oximeters overestimated arterial oxygen saturation more often in patients with darker skin. A widely cited 2020 New England Journal of Medicine analysis of paired oximeter and arterial-blood-gas readings found that Black patients had nearly three times the rate of "occult hypoxemia" (true saturation below 88 percent while the oximeter read 92 to 96 percent) compared with White patients. The consequence was not abstract: a reading in the reassuring range meant some patients were not flagged for supplemental oxygen or trial eligibility they in fact qualified for. The root cause traces to melanin's optical absorption biasing the red and infrared ratios the device converts to saturation, a transducer-level bias, compounded by historical calibration cohorts that under-represented dark skin. This is the archetype for the whole section: pooled device accuracy looked acceptable, the harm lived entirely in a subgroup, and the fix required changes at the sensor and calibration level, not a retrained classifier. In 2024 the FDA convened its advisory panel to tighten pulse-oximeter clearance to require skin-tone-diverse validation, an acknowledgment that demographic performance is a device-safety property.
From audit to fix, and what regulators now expect
Finding a gap obligates a response, and the response follows the source you diagnosed. If the gap is transducer-level, the honest fixes are hardware or per-group calibration and a documented restriction of the intended-use population, not a claim of parity you cannot support. If it is sampling, targeted collection to reach adequate subgroup counts, or reweighting, is the lever; simply oversampling in the loss without new data trades one group's error for another's and should be reported as such. If it is label bias, the reference standard itself must be re-examined. Whichever you choose, the operating threshold is a fairness knob: a single global cutoff on a score that is miscalibrated for one group will hand that group a worse error rate, so per-group threshold or calibration review belongs in the operating-point decision, tied to the calibration machinery of Chapter 18.
Regulators have moved from encouragement to expectation. The FDA's Good Machine Learning Practice principles call for datasets representative of the intended population and for performance reported across relevant subgroups, and its 2024 pulse-oximeter action shows the agency treating demographic performance as a clearance gate rather than a nice-to-have. The broader duty to design, monitor, and govern for equity across the whole sensing lifecycle is the subject of Chapter 70. The point that survives every regulatory revision is simple: a subgroup you did not measure is a subgroup you cannot claim to serve.
The Right Tool
The hand-rolled audit in Listing 34.6 covers one metric and one grouping. A real report needs many metrics, one- and multi-dimensional slices, intersectional cells, and consistent confidence intervals across all of them. Microsoft's fairlearn supplies this as a first-class object:
from fairlearn.metrics import MetricFrame, true_positive_rate, false_positive_rate
mf = MetricFrame(
metrics={"sensitivity": true_positive_rate, "fpr": false_positive_rate},
y_true=y, y_pred=p, sensitive_features={"skin_tone": grp})
print(mf.by_group) # per-group table for every metric
print(mf.difference()) # worst-case disparity per metricfairlearn replaces roughly 150 lines of grouping, metric, and disparity bookkeeping with a MetricFrame that reports every metric by group and the worst-case gap in one call, and extends to intersectional slices by passing multiple sensitive features. It computes the disparities; deciding which criterion bounds the clinical harm, and whether a gap is a sensor or a data problem, remains yours.The library removes the arithmetic, not the judgment. It will happily report a tight-looking disparity for a group of thirty patients; whether that estimate means anything is the sample-size question the tool cannot answer for you.
Exercise
You audit a wrist-wearable arrhythmia detector and find sensitivity of 0.91 (95% CI 0.88 to 0.93) on lighter skin and 0.79 (95% CI 0.64 to 0.90) on darker skin, with 55 diseased patients in the darker-skin group. First, state whether the gap is established or merely suspected, and justify your answer from the intervals. Second, propose one experiment that would tell you whether the cause is transducer-level or data-level, and name the different fix each cause implies.
Self-Check
1. Name the three upstream sources of bias in a health-sensing pipeline and give the distinct mitigation each one requires.
2. Why can calibration parity and equal false-positive rates be mathematically impossible to satisfy at once, and what decides which you should prioritize?
3. Why is a per-group sample size (and its confidence interval) essential to interpreting a fairness table, and what mistake follows from omitting it?
What's Next
In Section 34.7, we turn the audits, validation plans, and subgroup evidence of this chapter into durable artifacts: datasheets that describe the data, model cards that state the intended use and the demographic performance, and audit trails that let a regulator or a future engineer reconstruct exactly what was claimed, on whom, and why.