Adversarial robustness of detectors and the transductive-leakage trap

"I reported an F1 of 0.96. Then someone hid the answer key I had been peeking at, and my F1 fell to 0.41. The detector never changed; only my honesty did."

A Chastened Benchmark AI Agent

Attacking the detector, not just the plant

A residual detector is a differentiable function from sensor window to anomaly score, and anything differentiable can be optimized against. Suppose an attacker wants to drive a tank level to a damaging value while keeping the detector's score \(s(\mathbf{x})\) below its alarm threshold \(\tau\). If the attacker can perturb a subset of readings by \(\boldsymbol{\delta}\) (the sensors or actuators they control), the evasion is a constrained optimization,

where \(g\) measures the physical damage the attacker wants (the tank actually over-fills) and \(\epsilon\) bounds how far the spoofed readings may stray before other checks catch them. This is a physics-constrained adversarial example: unlike an image attack, the perturbation must respect that the plant is a coupled dynamical system, so the attacker perturbs the exogenous inputs and lets the process equations carry the lie downstream. The reason this works is the same reason the detector works at all. A neural residual model is locally smooth, so its score has usable gradients that point straight into the region where the process is damaged yet the reconstruction still looks normal. Autoencoders are especially soft targets because their generalization, the property that lets them reconstruct unseen normal regimes, also lets a gently nudged attack ride through the bottleneck with a small residual.

Three defenses raise the cost. Adversarial training augments the normal-only training set with worst-case perturbations of healthy windows so the score surface steepens around the manifold boundary. Randomized smoothing adds calibrated noise and thresholds an averaged score, converting a sharp exploitable cliff into a certified margin the attacker must overcome. And, most robust of all in cyber-physical settings, non-differentiable redundancy: the hard physical invariants of Section 38.1 (mass balance, actuator-state consistency) have no gradient for the attacker to descend, so an evasion that fools the learned scorer still has to satisfy conservation laws it cannot smoothly optimize around. A detector that is only a smooth learned function is a detector with a documented escape route; pairing it with an invariant checker of a fundamentally different mathematical type is what closes it.

The transductive-leakage trap

Now the self-inflicted wound, and the one that has quietly corrupted much of the published ICS-detection literature. A detector is inductive when every decision rule it applies at test time (its normalization statistics, its threshold, its model) is fixed from training-and-validation data before it ever sees a test sample. It becomes transductive, and leaky, the moment any of those rules is computed using the test window itself, because the test window contains the very attacks it is supposed to detect blind. Three habits do this, each looking innocent:

Test-set normalization. Min-max or z-score scaling fit on the whole evaluation stream folds the mean and spread of the attack samples into the transform every clean sample is judged against, so the attack has already tinted the ruler. The oracle threshold. By far the most damaging, this is the near-universal habit of choosing the alarm threshold \(\tau\) that maximizes F1 on the test labels themselves, reported as "best-F1." No deployed monitor can do this, because it presumes you already know which samples are attacks; the number it produces is an upper bound achievable only by an operator holding the answer key. Point-adjust scoring, introduced in Section 38.3, is the trap's accomplice: it counts a whole anomaly segment as caught if any single sample fires, and combined with an oracle threshold it lets a detector emitting essentially random scores post near-perfect F1. The fix is a discipline, not a trick: split chronologically, fit every statistic and threshold on a clean validation prefix, and freeze them before touching the test set, exactly the inductive contract of Chapter 65. Where you want the false-alarm rate to be a number you chose rather than one you hope for, calibrate the threshold conformally on clean validation residuals (Chapter 18).

import numpy as np
from sklearn.metrics import f1_score

# scores from a mediocre detector; higher = more anomalous
# y_test marks the true attack samples (unknown at deploy time)
def best_f1_oracle(scores, y):                 # TRANSDUCTIVE: peeks at labels
    taus = np.quantile(scores, np.linspace(0.5, 0.999, 200))
    return max(f1_score(y, scores > t) for t in taus)

def inductive_f1(s_val, s_test, y_test, q=0.995):   # honest: tau from clean val
    tau = np.quantile(s_val, q)                # fixed before seeing the test set
    return f1_score(y_test, s_test > tau)

rng = np.random.default_rng(0)
s_val  = rng.normal(0, 1, 5000)                            # clean validation
s_test = rng.normal(0, 1, 5000); y_test = np.zeros(5000, int)
atk = slice(2000, 2200); y_test[atk] = 1
s_test[atk] += rng.normal(0.6, 1, 200)         # weak, barely-separable attack

print(f"oracle 'best-F1'  = {best_f1_oracle(s_test, y_test):.2f}")
print(f"inductive F1      = {inductive_f1(s_val, s_test, y_test):.2f}")

Listing 38.3 makes the gap concrete and reproducible: swapping only the threshold-selection rule, with the identical detector and identical scores, moves the headline metric dramatically. When you read a paper reporting best-F1 with point-adjust, you are reading the oracle number, and the deployable performance is the lower, inductive one it usually omits.

from timeeval import TimeEval, DatasetManager, DefaultMetrics
# fits detectors, enforces a train/test boundary, and reports
# range-aware metrics WITHOUT the oracle point-adjust threshold
te = TimeEval(dm, algorithms, metrics=[DefaultMetrics.RANGE_PR_AUC])
te.run()