"My explanation was beautiful, the reviewer nodded, the operator agreed, and it pointed at the one channel the model never looked at. Everyone was happy except the physics."
A Persuasive AI Agent
Prerequisites
This section closes the chapter, so it assumes the attribution machinery of Section 67.1, the temporal saliency pitfalls of Section 67.2, and the counterfactual view of Section 67.3. It leans on the leakage-safe evaluation discipline of Chapter 65 (a benchmark that leaks is worse than no benchmark) and the synthetic-data-with-known-ground-truth idea from Chapter 55. We treat the explanation method as the object under test now, not the model: our job is to decide whether an attribution map tells the truth about \(f\).
The Big Picture
Every method in this chapter produces a confident-looking map. None of them come with a receipt proving the map reflects what the model actually did. That receipt is faithfulness: the degree to which an explanation reflects the true reasoning of the model it claims to explain. Faithfulness is distinct from plausibility, which is whether a human finds the explanation reasonable. A plausible-but-unfaithful map is the most dangerous artifact in interpretability, because it earns trust it has not paid for. This section gives you two tools to catch it: model-side perturbation metrics that need no ground truth, and XTSC-Bench, a benchmark built on synthetic time series where the true discriminative region is known by construction, so an explainer's precision and recall against ground truth can be measured directly rather than argued.
Faithful is not the same as plausible
The first mistake teams make is conflating two properties that pull in different directions. A clinician looking at an ECG saliency map that highlights the QRS complex will nod: that is where a cardiologist looks, so the map is plausible. But plausibility measures agreement with a human prior, not agreement with the model. If the network actually keyed on a baseline-wander artifact that happens to co-occur with the arrhythmia in the training set, the honest map should light up the artifact, and a map that instead lights up the QRS is unfaithful precisely because it looks right. Plausibility can therefore reward an explainer for lying convincingly. Faithfulness asks the harder question: if I intervene on the features the map calls important, does the model's output actually move the way the map predicts?
Adebayo and colleagues sharpened this with sanity checks: an explanation that does not change when you randomize the model's weights, or when you randomize the labels, cannot be reflecting the model, because it survives the destruction of the thing it claims to explain. Several popular saliency methods fail this test, producing near-identical maps for a trained network and for random noise. Any faithfulness protocol you adopt should include this cheap, brutal control before you trust a single downstream number.
Key Insight
Faithfulness is measured by intervention, not by inspection. The core move is: rank input features by the explanation, then delete (or insert) them in ranked order and watch the prediction. If the map is faithful, removing the top-ranked features should collapse the output fast; removing the bottom-ranked features should barely move it. This turns "is the explanation true?" into a curve you can plot and a scalar you can log, with no human in the loop and no ground-truth annotation required. The catch is entirely in the word delete: for a sensor stream there is no neutral value to delete a timestep to, which is the same baseline trap from Section 67.1 wearing an evaluation costume.
Perturbation metrics: measuring faithfulness without an oracle
The workhorse family needs no ground truth, only the model. Given an attribution map, sort the input positions from most to least important, then progressively perturb them and track how the predicted probability of the original class decays. Plotting that decay against the fraction of features perturbed gives a curve, and the area over it is the classic AOPC (area over the perturbation curve) or, equivalently, a deletion score. A steeply falling curve means the explanation identified the features the model truly relied on. Two symmetric variants sharpen the picture: deletion (start from the full input, remove top features, want a fast drop) and insertion (start from a baseline, add top features, want a fast rise). Reporting both guards against a method that games one direction.
For time series the perturbation operator is where faithfulness evaluation succeeds or quietly fails. Replacing a "removed" timestep with zero re-introduces the free-fall / asystole absurdity: you measure how sensitive the model is to an out-of-distribution cliff, not how important the feature was. Defensible operators replace the segment with the local mean, with Gaussian noise matched to the channel statistics, with a linear interpolation across the gap, or with a resample from a background of normal windows. The choice is a modeling decision that belongs in your evaluation report, not a hidden default, because different operators can reorder an explainer leaderboard.
import numpy as np
def deletion_faithfulness(model_prob, x, attribution, replace="mean", steps=20):
"""Deletion-style faithfulness for one window.
x: (C, T) array. attribution: (C, T) importance (higher = more important).
model_prob: callable (C, T) -> prob of the explained class.
Returns AOPC-style score in [0, 1]; higher = more faithful."""
order = np.argsort(-attribution.ravel()) # most important first
base = x.mean(axis=1, keepdims=True) * np.ones_like(x) if replace == "mean" \
else np.zeros_like(x) # the reference we delete TO
xp = x.copy().ravel()
ref = base.ravel()
p0 = model_prob(x) # prediction on full input
drops, k = [], max(1, len(order) // steps)
for i in range(0, len(order), k):
xp[order[i:i + k]] = ref[order[i:i + k]] # remove next chunk
drops.append(p0 - model_prob(xp.reshape(x.shape)))
return float(np.mean(drops)) # avg prob mass removed
replace argument: the whole validity of the number hinges on deleting to a physically meaningful reference (channel mean) rather than to zero. Averaging the probability drop over the deletion schedule yields the AOPC scalar referenced above.The routine above is the honest, transparent version so you can see the perturbation schedule and the reference choice. In practice you would not re-implement AOPC, faithfulness correlation, robustness, and the sanity-check randomizations by hand for every explainer.
Library Shortcut
The Quantus toolkit exposes faithfulness, robustness, localization, and randomization metrics behind one call each: quantus.FaithfulnessCorrelation()(model, x, y, a) returns the score, and swapping the class name gives you AOPC, monotonicity, or the model-randomization sanity check without touching the perturbation bookkeeping. That collapses roughly 30 to 40 lines of schedule, reference, and aggregation logic per metric down to a single constructor plus call, and it standardizes the operator choices so two teams report comparable numbers. For time-series-native explainers, TSInterpret ships the same evaluation surface next to the explainers it wraps, which we met as a review tool in Section 67.6.
XTSC-Bench: when the ground truth is known by construction
Perturbation metrics are model-faithful but self-referential: they tell you whether the map agrees with the model, not whether either agrees with reality. The complementary approach is to build a benchmark where the discriminative signal is planted deliberately, so the correct explanation is known. This is exactly what XTSC-Bench (Höllig, Kulbach, and Thoma) provides for time-series classification. It generates synthetic univariate and multivariate datasets in which a small, localized region (a bump, a step, a frequency burst) placed at a known channel and time interval is the only feature separating the classes; everything else is distractor noise. Because the informative region is labeled by the data generator, an explainer's map can be scored against a ground-truth mask with ordinary precision, recall, and rank metrics, rather than by human eyeballing.
XTSC-Bench then evaluates explainers along three axes that a single accuracy number would hide. Reliability asks whether the attribution actually lands on the planted region (localization against the mask). Robustness asks whether small, label-preserving perturbations of the input leave the explanation stable, since a map that reshuffles under imperceptible jitter is not measuring anything durable. Complexity (or sparsity) asks whether the explanation concentrates its mass or smears thinly across the whole window, because a map that highlights everything explains nothing. Crucially the benchmark holds the model and the training protocol fixed while varying the explainer, so differences are attributable to the explanation method rather than to a stronger classifier, and it standardizes the perturbation operators so that the deletion-reference trap above cannot silently swing the ranking.
Field story: a gearbox fault detector whose "best" explainer failed the mask test
An industrial team monitoring wind-turbine gearboxes (three vibration channels plus oil temperature, in the condition-monitoring spirit of Chapter 37) needs to attach a defensible explanation to every fault alert before a technician is dispatched. Two candidates look equally good in review: a gradient-saliency map and a perturbation-based explainer both produce crisp, plausible ribbons over the vibration channels, and reviewers prefer the saliency map because it is sharper. Before committing, the team rebuilds their pipeline on an XTSC-Bench-style synthetic set: a known frequency burst injected into channel 2 over a known 200 ms window, with the other channels pure distractors. Now ground truth exists. The sharp saliency map scores high on complexity (it is concentrated) but its reliability against the mask is poor: it repeatedly lands on channel 1, which co-varies with channel 2 but carries no planted signal, and it fails the robustness axis, jumping to a different channel under a 5% amplitude jitter. The less pretty perturbation explainer localizes on channel 2 and holds steady under jitter. The team ships the "uglier" method, and when a real fault later appears the attribution points a technician at the correct sensor and time on the first try. The benchmark did not make the model better; it stopped a persuasive-but-unfaithful explanation from reaching an operator.
Reading a faithfulness leaderboard without being fooled by it
Benchmarks earn trust by being used skeptically. First, no single scalar decides an explainer: reliability, robustness, and complexity trade off, and the right operating point depends on your consequence budget (a wrong dispatch versus a missed fault). Report the vector, not a scalar you invented by averaging incomparable axes, and co-compute the axes on one model, one split, one seed so the comparison is valid. Second, synthetic ground truth is necessary but not sufficient: an explainer that aces planted bumps may still fail on the entangled, multi-scale structure of real biosignals, so pair XTSC-Bench-style benchmarking with model-side perturbation metrics on your real data and with the sanity-check randomizations. Third, faithfulness interacts with calibration: attributions drawn from an over-confident model near its decision boundary are unstable, which is one more reason the calibration machinery of Chapter 18 is a prerequisite for trustworthy explanation, not a separate concern.
Research Frontier
Quantitative benchmarking of time-series explanations is young and consolidating fast. XTSC-Bench and the earlier synthetic-benchmark protocol of Ismail and colleagues established that many attribution methods that dominate on images degrade sharply on temporal data, especially as sequence length and the number of informative timesteps grow. Toolkits are converging: Quantus standardizes the metric side and TSInterpret the time-series explainer side, so an apples-to-apples comparison no longer requires re-implementing everyone's perturbation loop. The live open problems are the perturbation operator (there is still no consensus reference for "deleting" a sensor timestep), faithfulness metrics that respect temporal dependence rather than treating timesteps as exchangeable, and benchmarks whose synthetic generators capture the multi-scale, phase-jittered structure of real sensor streams instead of clean injected bumps. Treat today's leaderboards as directional, not final.
Exercise
Generate a synthetic three-channel classification set where a single 150 ms sinusoidal burst in channel 2 separates the two classes and the other channels are matched noise; record the ground-truth mask. Train a small classifier from Chapter 14. (1) Compute Integrated Gradients and a perturbation explainer, and score both with the deletion metric above using zero replacement and then mean replacement; report how the ranking changes. (2) Score both against the ground-truth mask with precision and recall. (3) Apply a 5% amplitude jitter and measure how much each map moves (robustness). State which explainer you would ship and cite the axis that decided it.
Self-Check
- Give a concrete sensor example of an explanation that is highly plausible yet unfaithful, and explain why plausibility failed to catch the problem.
- Why does the choice of "deletion reference" (zero vs channel mean vs background resample) change a faithfulness score, and how can it reorder an explainer leaderboard?
- XTSC-Bench reports reliability, robustness, and complexity separately. Why is collapsing them into one averaged number a mistake, and what does each axis independently protect you from?
Lab 67
Explain an anomaly alert with temporal attribution and nearest similar historical events; test faithfulness. Take a trained anomaly model on a real multivariate sensor stream, produce an attribution map plus the k nearest historical windows to the flagged event, then quantify the map with the deletion / insertion perturbation metrics under two references and with the model-randomization sanity check. Finally, port the pipeline to an XTSC-Bench-style synthetic set with a planted region and report precision, recall, robustness, and complexity against the known mask. Deliver a one-page faithfulness card that an operator could read before trusting the alert.
Bibliography
Faithfulness: definitions and sanity checks
Introduces the model-randomization and label-randomization controls: an explanation that survives destroying the model cannot be explaining it. The cheapest, most important test to run before trusting any map.
Origin of the perturbation-curve / AOPC methodology that underlies the deletion and insertion metrics used throughout this section.
Time-series faithfulness and benchmarks
Shows that image-tuned saliency methods degrade badly on temporal data and proposes synthetic benchmarks with known informative timesteps, the intellectual precursor to XTSC-Bench.
The section's namesake: synthetic datasets with planted discriminative regions and a reliability / robustness / complexity evaluation of time-series explainers against known ground-truth masks.
A perturbation-mask explainer designed for temporal saliency; a natural candidate to score on the faithfulness protocols above and a bridge to counterfactual masking.
Toolkits and reference implementations
Standardizes faithfulness, robustness, localization, and randomization metrics behind uniform calls, so teams report comparable numbers instead of bespoke perturbation loops.
Bundles time-series explainers with an evaluation surface, letting you generate and score attributions in one library; the practitioner counterpart to XTSC-Bench.
The axiomatic attribution method most often evaluated on these benchmarks; its completeness axiom is itself a faithfulness guarantee relative to a baseline.
What's Next
In Chapter 68, we turn from explaining a model's decisions to defending them: what happens when an adversary spoofs a sensor, how robustness and functional-safety standards frame acceptable failure, and why a faithful explanation of a spoofed input is the first line of forensic defense when a perception system is deliberately attacked.