"You asked which moment convinced me. I drew you a bright stripe. You never asked whether the stripe would move if I had learned nothing at all."
An Overconfident AI Agent
A heatmap over time is not evidence until it survives a sanity check
Attribution methods from Section 67.1 hand you a number for every sample of every channel. Collapse those numbers back onto the timeline and you get a temporal saliency map: a coloured stripe showing when the model "looked." It is the single most persuasive artifact in sensor interpretability, and the most dangerous, because a plausible stripe is trivially easy to produce and almost impossible to falsify by eye. Saliency over a time series is uniquely fragile: it must localize on two entangled axes at once (which instant, which channel), it inherits the discontinuities and saturation of the underlying network, and it can look identical whether the model learned physiology or memorized a firmware artifact. This section is about the failure modes and, more importantly, the tests that separate a saliency map that reflects the model's reasoning from one that is decorative noise.
This section assumes you can already produce attributions (Integrated Gradients, gradient times input, occlusion) from Section 67.1 and that you treat leakage as a first-class hazard from Chapter 5. Here we ask a narrower question: given a temporal saliency map, is it faithful (does it track what the model actually uses) and is it robust (does it stay put under trivial input changes)? Faithfulness and robustness are different properties, and a map can fail either.
Why time-series saliency is harder than image saliency
Saliency methods were largely developed and stress-tested on images, where a human can glance at a heatmap and judge whether it lands on the dog rather than the grass. Time series remove that safety net. Three structural differences make temporal saliency fragile.
No human prior on the raw signal. You can see a dog; you cannot see a QRS complex or a bearing-fault sideband in a wall of accelerometer samples. When the saliency stripe lands on sample 3,412 of a vibration trace, you have no independent way to say "yes, that is the informative moment." The reviewer's eye, the last line of defence for image saliency, is blind here.
Two axes of localization. A sensor window is a matrix of shape (time, channels). A faithful explanation must answer both when and which sensor, and the two interact. A gyroscope spike at heel-strike means one thing; the same spike in the magnetometer means a metal doorframe. Methods that sum attribution across channels to make a pretty one-dimensional stripe silently destroy the channel answer, and methods that treat every (time, channel) cell independently ignore that adjacent samples are almost the same measurement.
Temporal smearing and discontinuity. Raw gradients saturate: once a ReLU unit is firmly on or off, its local gradient is zero even though the feature was decisive, so a gradient map can go dark exactly where the model is most certain. Convolutional and attention receptive fields (Chapter 15) also blur attribution across neighbouring timesteps, so a single decisive event smears into a broad band and you cannot tell a 5-sample trigger from a 200-sample one.
A saliency map you cannot falsify is a picture, not a measurement
The core discipline of this section: never trust a saliency map you have not tried to break. Two cheap adversarial tests catch most fraud. The model-randomization test re-runs the exact same saliency method on a network with randomly reinitialized weights; if the map barely changes, the method is drawing the input, not the model, and it is worthless as an explanation. The perturbation test deletes the samples the map calls important and checks that the prediction actually collapses; if it does not, the map is unfaithful. A stripe that survives both is a measurement. A stripe that survives neither is decoration.
Sanity checks: model and data randomization
Adebayo and colleagues showed that several popular saliency methods produce visually convincing maps that are invariant to the model: randomize the weights layer by layer and the heatmap is nearly unchanged. Such a method is effectively an edge detector on the input; it would draw the same stripe for a trained network, an untrained one, and a network trained on scrambled labels. For sensor data this is not a curiosity, it is the default risk, because the input itself has salient structure (peaks, transients) that any input-driven method will latch onto regardless of what the model learned.
Two randomization tests form the minimum bar. The model-parameter randomization test compares the saliency on the trained model against the saliency on the same architecture with reinitialized weights; a faithful method should decorrelate sharply. The data-randomization test retrains on permuted labels and re-checks; a method that produces the same map on a model that has learned nothing meaningful is not reading the model. Report the rank correlation between the real and randomized maps as a number, not a side-by-side picture, because the eye forgives far too much.
import numpy as np
from scipy.stats import spearmanr
from captum.attr import IntegratedGradients
def temporal_saliency(model, x, target):
ig = IntegratedGradients(model)
a = ig.attribute(x, target=target, n_steps=64) # (batch, time, channels)
return a.abs().sum(dim=-1).squeeze().cpu().numpy() # collapse channels -> (time,)
def model_randomization_test(build_model, trained, x, target, n=10):
"""Faithful saliency should NOT survive weight randomization."""
real = temporal_saliency(trained, x, target)
corrs = []
for _ in range(n):
rand = build_model() # fresh random weights
s = temporal_saliency(rand, x, target)
rho, _ = spearmanr(real, s) # rank correlation of maps
corrs.append(rho)
return real, float(np.mean(corrs)) # want |mean rho| near 0
# rho near 0.9 => the map ignores the model; reject the method for this task
The snippet above uses Captum's Integrated Gradients so the attribution itself is one call; the test around it is what earns trust. Run it before any saliency map reaches a slide.
Perturbation-based faithfulness: deletion and insertion
Sanity checks catch model-blind maps; they do not confirm that the highlighted samples cause the prediction. For that, perturb. The deletion curve removes samples in order of decreasing saliency (replacing them with a neutral baseline, usually the per-channel mean or a linear interpolation) and records the class score after each removal; a faithful map makes the score drop fast, giving a small area under the deletion curve. The insertion curve does the reverse, starting from an empty baseline and adding the most-salient samples first; faithful maps recover the score quickly, giving a large area under the insertion curve. Reporting both guards against the artifact where deletion "works" only because blanking any samples creates an out-of-distribution input the model dislikes for reasons unrelated to importance.
The perturbation choice is not innocent. Replacing a decisive transient with the channel mean can create a step discontinuity that the model reacts to as a new event, so the deletion drop overstates faithfulness. Prefer perturbations that stay on the data manifold: local interpolation, phase-preserving noise, or resampling from a matched baseline segment. This is the same manifold discipline that governs counterfactual explanations in Section 67.3, and it connects to distribution-shift robustness in Chapter 66: an off-manifold perturbation measures the model's OOD reflex, not its use of the feature.
A bearing-fault detector whose saliency pointed at the tachometer, not the bearing
An industrial team building a rolling-element bearing fault classifier (see Chapter 37) shipped a slick temporal saliency dashboard: for every alarm it drew a stripe over the vibration window. Engineers loved it until a maintenance shift noticed the stripe always brightened at the same phase of the shaft rotation regardless of fault type. A model-randomization test settled it: the map's Spearman correlation with a randomly initialized network was 0.88, so the saliency was tracking the periodic tachometer-locked structure of the input rather than the fault. A deletion test confirmed the betrayal: removing the "important" samples barely moved the score, while removing the true fault sidebands (which the map had ignored) collapsed it. The root cause was leakage, the machine's rotation speed correlated with the labelled test rig, and the saliency had faithfully visualized a shortcut. The stripe was real; what it explained was the wrong thing. The fix was a leave-machine-out split and a saliency method that passed both sanity checks before any dashboard was built.
TSInterpret runs the whole faithfulness battery for you
Hand-rolling deletion and insertion curves, on-manifold perturbations, and the randomization comparison is roughly 150 to 200 lines of careful code, and the subtle bugs (baseline leakage, off-by-one masking, forgetting to renormalize the score) all inflate faithfulness. TSInterpret exposes these as configured evaluators: build the explainer, then call its perturbation-faithfulness and robustness metrics in a handful of lines, with time-series-aware baselines already implemented. You still choose the perturbation and read the numbers critically; the library removes the plumbing, not the judgement. We return to TSInterpret and Captum as review tooling in Section 67.6, and to standardized faithfulness benchmarks in Section 67.7.
Robustness: the same input, a slightly different map
A faithful map can still be untrustworthy if it is unstable. Add imperceptible noise, shift the window by one sample, or nudge the integration baseline, and a fragile saliency method will redraw the stripe somewhere else while the prediction is unchanged. Formally, we want the explanation map \(E\) to be Lipschitz in the input: for small \(\lVert x' - x \rVert\), we want \(\lVert E(x') - E(x) \rVert\) small too. Estimate it empirically as a local stability score, $$ \hat{L}(x) = \max_{x' : \lVert x'-x \rVert \le \epsilon} \frac{\lVert E(x') - E(x) \rVert_2}{\lVert x' - x \rVert_2}. $$ High \(\hat{L}\) means the map moves faster than the input, so any single stripe you show is one sample of a jittery distribution. Report stability alongside faithfulness; a method can be faithful in expectation yet so unstable that no individual map should be trusted. Averaging attributions over a small noise cloud (the SmoothGrad idea) trades a little sharpness for a large gain in stability and is usually worth it for sensor streams, where sampling jitter and synchronization error (Chapter 3) guarantee that the exact sample alignment you attributed will never recur in the field.
Break your own saliency map
Take a trained ECG arrhythmia classifier (or any sensor model you have from Chapter 29) and one correctly classified window. (1) Produce a temporal saliency map with Integrated Gradients. (2) Run the model-randomization test from the code block over 10 reinitializations and report the mean Spearman correlation. (3) Build a deletion curve using two baselines, per-channel mean and local linear interpolation, and report both areas under the curve. (4) Add Gaussian noise at 1 percent of the signal amplitude, recompute the map, and report its correlation with the original. Which of the three failure modes (model-blindness, off-manifold deletion, instability) does your setup suffer from, and does SmoothGrad averaging fix the one you found?
Self-check
- A colleague shows a beautiful saliency stripe as proof the model learned physiology. What single number would you ask for first, and what value would make you reject the claim outright?
- Why can a deletion curve overstate faithfulness when you replace removed samples with the channel mean, and what perturbation would you use instead?
- Faithfulness and robustness are distinct. Give one method that is faithful on average yet untrustworthy per-map, and name the metric that would expose it.
What's Next
In Section 67.3, we move from highlighting what the model used to changing it: counterfactual sensor explanations (TSEvo, MASCOTS) that answer "what minimal, on-manifold edit to this signal would have flipped the decision?" The manifold discipline we demanded of deletion perturbations here becomes the central design constraint there, and the same faithfulness instinct carries over: a counterfactual is only useful if the edit is realistic and the flip is real.