Honest evaluation (TSB-AD, point-adjust pitfalls)

"My detector scored 0.96 F1 on the benchmark. Then I replaced it with a coin flip and scored 0.97. The coin flip was cheaper, so I shipped that."

A Chastened AI Agent

Honest evaluation here rests on the leakage-safe splitting discipline of Chapter 5 and the general benchmarking protocols of Chapter 65; this section is the anomaly-detection-specific instance of those rules. It also closes a loop opened in Section 37.4, where we built residual-to-alarm pipelines but deferred the reckoning about how to score them, and it extends the point-adjust warning first raised for classical detectors in Chapter 12. Prerequisites are light: precision, recall, and the precision-recall curve from any first ML course, plus the notion of a calibrated score threshold from Chapter 18.

The point-adjustment trap

Anomalies in machine data are almost always segments, not isolated points: a cavitating pump stays cavitated for hundreds of samples. A detector emits a per-sample score, you threshold it, and you would like credit for "catching the event" even if you did not flag every single sample inside it. Point-adjustment (PA), introduced with the Donut and OmniAnomaly detectors around 2018, encodes that wish as a rule: if the detector flags at least one sample inside a ground-truth anomaly segment, mark the entire segment as correctly detected. Then compute ordinary point-wise precision, recall, and F1 on the adjusted predictions.

The intent is defensible; the arithmetic is not. Consider a segment of length \(L\). Under PA, a single true-positive sample is promoted to \(L\) true positives. As anomaly segments in the popular benchmarks are long (tens to hundreds of samples), recall becomes almost free: any score that occasionally pokes above threshold inside a long segment harvests the whole thing. Because the benchmarks are also heavily imbalanced, precision is dominated by those inflated true positives, and F1 rockets toward 1. The pathology is that PA rewards hitting a segment anywhere far more than it penalizes flooding the normal regime with false alarms, which is precisely the tradeoff a condition-monitoring system must get right.

Watch a coin flip win

The fastest way to internalize this is to run it. The code below builds a synthetic monitoring trace with 40 anomaly segments, defines a useless detector that flags 5% of samples uniformly at random, and reports its F1 before and after point-adjustment. Raw F1 sits near chance; the point-adjusted F1 leaps into the range that gets papers accepted.

import numpy as np

def point_adjust(pred, label):
    """If any flagged sample lands inside a true anomaly segment,
    promote the whole segment to 'detected' (the standard PA rule)."""
    pred = pred.copy()
    in_seg, start = False, 0
    for i in range(len(label)):
        if label[i] and not in_seg:
            in_seg, start = True, i
        if not label[i] and in_seg:
            if pred[start:i].any():
                pred[start:i] = 1
            in_seg = False
    if in_seg and pred[start:].any():
        pred[start:] = 1
    return pred

def f1(pred, label, eps=1e-9):
    tp = np.sum((pred == 1) & (label == 1))
    fp = np.sum((pred == 1) & (label == 0))
    fn = np.sum((pred == 0) & (label == 1))
    p, r = tp / (tp + fp + eps), tp / (tp + fn + eps)
    return 2 * p * r / (p + r + eps)

rng = np.random.default_rng(0)
T, label = 20_000, np.zeros(20_000, dtype=int)
for s in rng.integers(0, T - 60, size=40):        # 40 anomaly events
    label[s:s + rng.integers(30, 60)] = 1

pred = (rng.random(T) > 0.95).astype(int)          # a USELESS detector

print("raw F1          :", round(f1(pred, label), 3))
print("point-adjusted F1:", round(f1(point_adjust(pred, label), label), 3))
# raw F1           : 0.05
# point-adjusted F1: 0.86

Metrics that survive scrutiny

Rejecting PA does not mean reverting to naive point-wise F1, which is too harsh in the opposite direction: it demands you flag every sample of an event and gives no partial credit for a slightly early or slightly late detection. The field has converged on a small set of range-aware alternatives, and you should report at least one from each idea below.

Range-based precision and recall (Tatbul et al., 2018) generalize the point-wise definitions to intervals, with tunable terms for existence (did you touch the event at all), size (how much of it), and position (front-biased credit when early detection matters, which it usually does in prognostics). Affiliation metrics (Huet et al., 2022) score the average temporal distance between predicted and true events and normalize against a random baseline, so a detector must beat chance by construction, closing the exact loophole PA leaves open. Threshold-free area metrics avoid the separate sin of tuning the alarm threshold on the test set: instead of one operating point, report the area under the precision-recall curve (preferred over ROC because anomalies are rare and PR is not fooled by the enormous true-negative mass), or its range-aware generalization VUS (Volume Under the Surface, Paparrizos et al., 2022), which integrates over both the threshold and a temporal tolerance so no single lucky cutoff can be cherry-picked. When you must report a single-threshold number, use PA%K, which only credits a segment if at least \(K\%\) of its samples are flagged and sweeps \(K\) to expose the sensitivity that plain PA hides.

For condition monitoring specifically, add two operational numbers no leaderboard captures: detection delay (samples or seconds from fault onset to first alarm, because a correct alarm that arrives after the failure is worthless) and false-alarm rate per machine-day (the quantity that decides whether operators keep the system switched on). The calibration machinery of Chapter 18 turns a raw residual into a threshold that targets that false-alarm rate directly, rather than an arbitrary quantile.

TSB-AD and honest benchmarking

Fixing the metric is necessary but not sufficient, because several classic benchmarks are themselves flawed: they contain mislabeled anomalies, trivially detectable single spikes, and train/test contamination that a leakage audit would reject. TSB-AD (Time-Series Benchmark for Anomaly Detection, Liu and Paparrizos, NeurIPS 2024) is the community's response. It curates over a thousand time series across univariate and multivariate domains, manually screens out mislabeled and trivial cases, standardizes on threshold-free VUS and range-aware metrics rather than point-adjusted F1, and evaluates dozens of statistical, machine-learning, and deep methods under one protocol. Its headline finding is deflating and useful: once the metric is fixed, simple, well-tuned methods (nearest-neighbor distances, sub-sequence clustering, classical forecasting residuals) are competitive with or ahead of many elaborate deep models, echoing the "model normal, score the residual" pragmatism of Section 37.4. TSB-AD is the benchmark to report against for a new detector, and its leaderboard is the sanity check when your in-house number looks too good.

What's Next

In Chapter 38, we push anomaly detection from "a machine is degrading" to "a machine is under attack." When the abnormal readings are produced by an adversary rather than a worn bearing, the physics-based process invariants, the attack-versus-fault distinction, and the evaluation protocols all change, and the honest-metric discipline you just learned becomes a security requirement rather than a scientific nicety.