Task metrics vs operational metrics

"My F1 score was 0.97. The night shift still turned me off. Nobody in the report explained why those two facts were allowed to coexist."

A Decommissioned AI Agent

Why a task metric can lie about the field

A task metric summarizes label separation on a held-out set under an implicit set of assumptions: the test distribution matches deployment, every error costs the same, and the operating threshold is whatever maximized the metric. Sensor systems violate all three routinely. The consequence is a number that is true about the test file and false about the running product.

Consider the arithmetic that a threshold-free score hides. Fall detection on a wrist wearable might report an AUROC of \(0.98\), which sounds like a solved problem. But falls are rare. Suppose a genuine fall occurs about once per user-month while the device evaluates a candidate window every few seconds, so the negative class outnumbers the positive by something like \(10^{6}\) to one. Even a very good operating point with a false-positive rate of \(0.1\%\) then produces roughly \(10^{6} \times 0.001 = 1000\) false alarms for each real fall. The AUROC did not lie about ranking; it simply never mentioned that the base rate turns a small error rate into a wall of nuisance alerts. The operational metric that exposes this is the precision at the deployed threshold, or equivalently the false-alarm count per user per week, and it is the number the product team actually lives with. This base-rate trap returns in force in Section 65.4.

The general move is to compute the metric the user experiences, not the one the loss function optimized. That almost always means fixing a real operating point (a threshold, a staffing budget, a latency ceiling) and reading off the consequence in the units the operator counts.

Building the bridge: operating points and cost curves

The bridge from task to operations has three planks. First, fix the operating point the way the deployment will. If the on-call team can absorb five alerts per shift, then the threshold is whatever yields five alerts per shift on representative data, and you report recall at that budget, not the recall that maximizes F1. Second, attach a cost to each cell of the confusion matrix. Let \(c_{\text{FP}}\) be the cost of a false alarm (a wasted truck roll, an unnecessary clinical review) and \(c_{\text{FN}}\) the cost of a miss (an unplanned outage, a missed arrhythmia). The expected operational cost per window at threshold \(\tau\) is

where \(\pi\) is the event base rate. The optimal threshold minimizes this quantity, and it is generally nowhere near the F1-maximizing point when \(c_{\text{FN}} \gg c_{\text{FP}}\) or when \(\pi\) is tiny. Third, express latency and throughput as first-class metrics. A predictive-maintenance model that flags a bearing failure two hours before it happens is worthless if the crew needs a two-day lead time to source the part; the operational metric is not "did it detect the fault" but "did it detect the fault with enough lead time to act," a horizon that connects directly to the prognostics work in Chapter 36.

What to report: paired task and operational metrics

The discipline is simple to state and easy to skip: never report a task metric alone. Report it paired with the operational metric that shares its operating point, computed in one pass on one configuration so the two numbers are genuinely about the same decision. A leaderboard row for a sensor model should read like a small table, not a single headline. AUROC pairs with "false alarms per device per week at the chosen threshold." Mean absolute error on a regression pairs with "fraction of predictions inside the tolerance band the controller can act on." Detection latency pairs with "lead time relative to the action deadline." The pairing is what makes the row actionable and what makes leakage-inflated numbers, the subject of the rest of this chapter, harder to hide behind.

import numpy as np

def operational_report(scores, labels, cost_fp, cost_fn, base_rate,
                       alerts_budget_per_day, windows_per_day):
    """Pick the threshold that meets an alert budget, then report
    the task metric AND the operational consequence at that point."""
    order = np.argsort(-scores)                 # rank by score, high first
    s, y = scores[order], labels[order]
    # Threshold that yields the allowed number of alerts per day
    frac_allowed = alerts_budget_per_day / windows_per_day
    k = max(1, int(frac_allowed * len(s)))
    tau = s[k - 1]
    pred = scores >= tau
    tp = np.sum(pred & (labels == 1)); fp = np.sum(pred & (labels == 0))
    fn = np.sum(~pred & (labels == 1))
    recall = tp / max(tp + fn, 1)               # task metric at this point
    precision = tp / max(tp + fp, 1)
    fpr = fp / max(np.sum(labels == 0), 1)
    exp_cost = (cost_fp * (1 - base_rate) * fpr
                + cost_fn * base_rate * (1 - recall))   # operational metric
    return dict(threshold=float(tau), recall=round(recall, 3),
                precision=round(precision, 3),
                alerts_per_day=round(fpr * (1 - base_rate) * windows_per_day, 1),
                expected_cost=round(float(exp_cost), 4))

rng = np.random.default_rng(0)
labels = (rng.random(200_000) < 0.01).astype(int)        # 1% base rate
scores = np.clip(rng.normal(labels * 1.6, 1.0), -5, 6)   # noisy detector
print(operational_report(scores, labels, cost_fp=1, cost_fn=200,
                         base_rate=0.01, alerts_budget_per_day=20,
                         windows_per_day=200_000))

Code 65.1.1 makes the pairing mechanical: the threshold is chosen by the budget, and every returned number is read at that threshold. Swapping in the true \(c_{\text{FP}}\), \(c_{\text{FN}}\), and base rate for your domain turns the last field into a decision-grade quantity rather than a benchmark trophy.

When the two metrics diverge, trust the operational one

A healthy divergence between task and operational metrics is diagnostic, not embarrassing. When AUROC is high but alerts-per-shift is intolerable, the model ranks well and the base rate is punishing: the answer is a persistence rule, a higher threshold, or a two-stage cascade, not a bigger network. When mean absolute error is low but the controller still oscillates, the errors are concentrated exactly where the tolerance band is tight: report error conditioned on the operating region. When latency is the binding constraint, no accuracy gain matters until the model fits the deadline, which is why the edge-deployment chapters (Chapter 60) treat latency budgets as a first-class evaluation axis. In every case the operational metric is closer to the money and the risk, so when the two disagree, it is the operational number that decides whether to ship.

What's Next

In Section 65.2, we turn from which metrics to report to how to split the data so those metrics are honest. A beautiful paired report is still a fiction if the same user, site, device, or session leaks across the train and test folds, and we will see why the naive random split silently inflates every number this section taught you to compute.