Rare-event and imbalanced evaluation

"My fall detector scored 99.8% accuracy. It had never once detected a fall. It had simply learned that people, mostly, do not fall."

A Chastened AI Agent

Why accuracy and ROC quietly lie

Class imbalance is the rule, not the exception, in sensory AI. Prevalence, the fraction of samples that are positive, is often \(10^{-2}\) for wearable falls, \(10^{-4}\) for individual anomalous vibration windows, and lower still for security events. Two widely quoted metrics fail silently at these rates. Accuracy is dominated by the majority class: at prevalence \(\pi\), the trivial always-negative classifier scores \(1-\pi\), so a 99.9% headline number can mean the detector has never fired. ROC-AUC is subtler and more dangerous because it looks rigorous. The ROC curve plots true-positive rate against false-positive rate, and the false-positive rate has the negative count in its denominator. When negatives outnumber positives ten-thousand-to-one, an absolute flood of false alarms barely moves the false-positive rate, so ROC-AUC stays near 1.0 while an operator drowns in false positives. ROC-AUC is invariant to prevalence by construction, which is precisely why it cannot tell you whether the system is deployable.

The imbalance-aware alternative is the precision-recall view. Precision (also called positive predictive value) is \(\mathrm{TP}/(\mathrm{TP}+\mathrm{FP})\): of the alarms you raised, what fraction were real. Recall (sensitivity) is \(\mathrm{TP}/(\mathrm{TP}+\mathrm{FN})\): of the real events, what fraction you caught. Neither denominator contains the huge true-negative count, so precision-recall analysis reacts sharply to false alarms that ROC ignores. The area under the precision-recall curve (PR-AUC, or average precision) is the imbalance-aware summary statistic, and its no-skill baseline is not 0.5 but the prevalence \(\pi\) itself, which keeps you grounded about how hard the problem is.

Score the events, not the samples

Rare events in sensor streams are almost never single samples; they are intervals. A fall spans a second of accelerometer data across dozens of windows, a fault develops over minutes, an apnea event lasts tens of seconds. Scoring each window independently, the naive point-wise approach, produces two artifacts that make the report meaningless. First, it inflates the positive count: one real event becomes forty positive windows, so a model that catches the middle of every event but misses the edges looks like it has poor recall when operationally it caught everything that mattered. Second, it undercounts false alarms in operational terms: forty consecutive false-positive windows are one nuisance alarm to a human operator, not forty. The metric and the cost function have drifted apart, exactly the failure Section 65.1 warns against.

Event-based evaluation fixes this by matching predicted segments to ground-truth segments before counting. A predicted alarm that overlaps a true event (by any overlap, or by a required minimum intersection, depending on the protocol) counts as one true positive; a true event with no overlapping alarm is one false negative; an alarm overlapping nothing is one false positive. On top of the event-level confusion matrix, two operational quantities become the currency of rare-event work. The false-alarm rate is expressed per unit time, alarms per hour or per device-day, because "how often will this wake someone at 3 a.m." is what a customer actually asks. Detection latency, the delay between event onset and the first correct alarm, matters whenever early warning is the point, as in the change-detection methods of Chapter 12 and the condition monitoring of Chapter 37. A report that gives recall, precision, false alarms per device-day, and median latency describes a deployable system; a point-wise F1 alone does not.

How sure are you with twelve positives?

Rarity does not only distort the point estimate; it wrecks its precision. If your held-out test set contains twelve true events and the model catches nine, the reported recall is 0.75, but that estimate rests on twelve Bernoulli trials. Its 95% confidence interval, by the Wilson or Clopper-Pearson method, stretches from roughly 0.43 to 0.93. A competing model that scores 0.83 on the same twelve events is very likely inside the noise, and declaring it the winner is measuring the coin, not the model. Every rare-event metric needs an error bar, and the width of that error bar is governed by the number of positives, not the total sample count. Ten million negatives do nothing to sharpen your recall estimate.

The general-purpose tool is the bootstrap: resample the evaluation set with replacement many times, recompute the metric on each resample, and read the empirical 2.5th and 97.5th percentiles as the interval. For event-level metrics the resampling unit must be the event or the recording, not the sample, or you fabricate confidence you do not have; this mirrors the clustered-split discipline of Section 65.2. The practical consequence is a design constraint felt long before evaluation: to distinguish two detectors at the precision you want, you need enough positive events in the test set, and with a rare target that can mean collecting for months. Reporting a bare 0.75 with no interval is not conservative; it is a claim of certainty you have not earned. The complementary question of whether the model's predicted probabilities are trustworthy, rather than just its thresholded decisions, is taken up as calibration in Section 65.6 and Chapter 18.

import numpy as np
from sklearn.metrics import average_precision_score, roc_auc_score

rng = np.random.default_rng(0)
n, prevalence = 100_000, 0.001            # 100 true positives among 100k
y = (rng.random(n) < prevalence).astype(int)
# A mediocre detector: scores overlap heavily between classes.
scores = rng.normal(0.0, 1.0, n) + 1.2 * y

print(f"positives           : {y.sum()}")
print(f"ROC-AUC   (flatters): {roc_auc_score(y, scores):.3f}")
print(f"PR-AUC    (fair)    : {average_precision_score(y, scores):.3f}")
print(f"no-skill PR baseline: {y.mean():.3f}")   # equals the prevalence

# Bootstrap a 95% interval for PR-AUC, resampling whole samples here for brevity.
boot = [average_precision_score(y[i], scores[i])
        for i in (rng.integers(0, n, n) for _ in range(1000))]
lo, hi = np.percentile(boot, [2.5, 97.5])
print(f"PR-AUC 95% CI       : [{lo:.3f}, {hi:.3f}]")

Listing 65.4.1 makes the gap concrete: the ROC number and the PR number describe the same predictions, but only the second reacts to the class imbalance the deployment will actually face, and only the interval tells you whether a difference between two detectors is real.

from sklearn.metrics import average_precision_score, precision_recall_curve
ap = average_precision_score(y, scores)               # PR-AUC in one line
precision, recall, thresholds = precision_recall_curve(y, scores)

What's Next

In Section 65.5, we hold the metrics steady and start breaking the sensors: what a benchmark must do to earn trust when a channel drops out, a signal saturates, or a device degrades mid-recording. Rare-event evaluation and missing-sensor robustness meet in the field, where the alarm you most need is often the one raised while an input is quietly failing.