OOD and novelty detection

"I was trained to name five gestures. The sixth one, I named with total confidence anyway. That was the moment they should have stopped trusting me."

An Overconfident AI Agent

OOD, novelty, and anomaly: three names, one gate

The literature uses three words that overlap enough to confuse and differ enough to matter. Anomaly detection (Chapter 12) flags a sample as unusual relative to a mostly-normal stream, and classically it works on a single signal with no notion of learned classes. Novelty detection flags an input that belongs to a genuinely new category, the sixth gesture, a bearing fault never before catalogued. OOD detection is the machine-learning framing: an input drawn from a distribution different from the training distribution \(p_{\text{train}}(x)\), whether that is a new class (semantic shift) or the same classes under new conditions (covariate shift). For a deployed sensor model the three collapse into one operational gate: before I act on this prediction, does the input fall inside the region where my model was ever validated? Everything else in this section builds that gate.

Two regimes deserve separate names because they demand different scores. Far-OOD is an input from an obviously foreign source: an accelerometer stream fed into an ECG model, a sensor unplugged and reading rail noise. Near-OOD is the hard and common case in sensing: the same modality, same subject population, but a new activity, a new device firmware, a new mounting position. Vision benchmarks are dominated by far-OOD (CIFAR versus SVHN); sensor deployments are dominated by near-OOD, and the scores that ace the former routinely fail the latter.

Score functions: from softmax to feature geometry

The cheapest OOD score is the maximum softmax probability (MSP): \(s_{\text{MSP}}(x) = \max_k \, p_k(x)\). It needs no extra training and often works for far-OOD. Its weakness is structural: modern networks are overconfident off the training manifold, so a foreign input can still produce a spiky softmax. The energy score repairs much of this by reading the logits before the softmax normalizes away their scale,

where \(z_k\) are the logits and \(T\) is a temperature. Lower total logit mass means lower energy for ID inputs and higher energy for OOD ones, and unlike MSP it is not squashed into \([0,1]\), so it preserves the confidence gap that the softmax destroys. A third family ignores the classifier head entirely and works in the penultimate feature space \(f(x)\). The Mahalanobis score fits a class-conditional Gaussian to the training features and scores a test point by its distance to the nearest class mean under the shared covariance \(\Sigma\),

and its non-parametric cousin, the k-nearest-neighbour score, uses the distance to the \(k\)-th closest training feature with no Gaussian assumption at all. The feature-space scores tend to win on the near-OOD cases that matter for sensors, because a new activity lands off the training clusters in embedding space even when its logits look ordinary. The quality of these scores rides entirely on the representation, which is why the self-supervised and contrastive encoders of Chapter 17 give better OOD separation than a representation trained only to minimize cross-entropy.

Setting the threshold without ever seeing the outliers

A score is only half a detector; the other half is a threshold \(\tau\), and the defining constraint of real deployment is that you cannot tune \(\tau\) on OOD data because you do not have a representative sample of it. The honest procedure sets \(\tau\) on ID validation data alone to hit a chosen true-positive rate on the ID class, for example the threshold at which \(95\%\) of legitimate windows are accepted. The headline number is then the false-positive rate at 95% TPR (FPR95): of the OOD inputs, what fraction slip through the gate as if they were ID. Reporting AUROC alongside FPR95 is standard, but FPR95 is the operational quantity, because it maps directly to how often a novel input reaches your classifier unchallenged. This is the same discipline as the operating-point argument in Chapter 65: a threshold-free score flatters you, a fixed operating point tells you the truth. Two cautions specific to sensing. First, calibrate \(\tau\) on a held-out validation stream that respects subject and session boundaries, or leakage will make the gate look far tighter than it is. Second, the score distribution drifts with the same covariate shifts this chapter studies, so \(\tau\) is not a one-time constant; Section 66.6 returns to monitoring it online.

import numpy as np

def energy_score(logits, T=1.0):
    """Free-energy OOD score: lower for OOD, higher for in-distribution."""
    return T * np.log(np.exp(logits / T).sum(axis=1))   # logsumexp over classes

def fit_threshold(id_scores, tpr=0.95):
    """Threshold that ACCEPTS `tpr` of in-distribution windows. OOD never seen."""
    return np.quantile(id_scores, 1.0 - tpr)

def fpr_at_threshold(ood_scores, tau):
    """Fraction of OOD windows that leak through the gate as in-distribution."""
    return float(np.mean(ood_scores >= tau))

rng = np.random.default_rng(0)
K = 6                                              # six known activities
id_logits  = rng.normal(0.0, 1.0, (4000, K)); id_logits[:, 0] += 4.0   # excited, correct
ood_logits = rng.normal(0.0, 1.0, (4000, K))       # novel activity: no class excited

id_s, ood_s = energy_score(id_logits), energy_score(ood_logits)
tau = fit_threshold(id_s, tpr=0.95)
print(f"tau={tau:.2f}  FPR95={fpr_at_threshold(ood_s, tau):.3f}")

Code 66.2.1 is deliberately minimal to expose the logic: the threshold never touches the OOD array, and the reported FPR95 is the number a safety reviewer will ask for. Swapping energy_score for a Mahalanobis or kNN score on penultimate features is a drop-in change and usually the right move for near-OOD sensor data.

When OOD detection is the wrong tool

A gate that fires too often is a decommissioned gate. If your near-OOD is really covariate shift you could adapt to (a new device, a new wearing position), then refusing every shifted window wastes information the test-time adaptation of Sections 66.4 and 66.5 could exploit instead. The design question is whether the shift is recoverable (same task, adaptable features) or semantic (a genuinely new class you must not classify). OOD detection is the right response to the second; domain generalization and adaptation are the right response to the first. Confusing them produces either a brittle classifier that hallucinates classes or a paranoid gate that escalates everything, and the whole point of this chapter is to give you the vocabulary to tell them apart before you pick a mechanism.

What's Next

In Section 66.3, we turn from detecting the shift to surviving it. Instead of gating out every unfamiliar window, domain generalization trains a model whose representation is stable across environments in the first place, and we will meet DIVERSIFY, a method that discovers latent domains inside a single sensor dataset and forces the network to generalize across them before deployment ever begins.