Section 67.1: Attribution for time series (Integrated Gradients, SHAP-TS)

"I assigned all the blame to the very first millisecond, because that is where I decided to start counting from. Nobody asked me where zero should be."
An Overconfident AI Agent

Prerequisites

This section assumes the gradient-based training vocabulary of Chapter 13 and the sequence architectures whose predictions we are about to explain, especially the attention models of Chapter 15. Basic multivariate notation for a windowed sensor tensor comes from Chapter 3, and the expectation and baseline-distribution language from the probability primer in Chapter 4. We treat a trained model $f$ as given: this section is not about building it, it is about explaining a single one of its predictions by distributing credit across the input samples that produced it.

The Big Picture

An attribution method takes one prediction $f(x)$ on one input window $x$ and returns a number for every input coordinate, telling you how much that coordinate contributed to the output. For a sensor window $x \in \mathbb{R}^{C \times T}$ (C channels, T timesteps) that means an attribution map of the same shape: a heat value on every (channel, timestep) cell. Two methods dominate practice because both rest on axioms rather than intuition. Integrated Gradients integrates the model's gradient along a straight path from a baseline to the input and provably satisfies completeness: the attributions sum to $f(x) - f(\text{baseline})$. SHAP assigns each coordinate its Shapley value, the unique credit allocation satisfying a set of game-theoretic fairness axioms. For time series the hard part is not the algorithm, it is the two choices the algorithm hides: what baseline represents "absence" of a signal, and at what granularity credit is meaningful. Get those wrong and the map is confidently, axiomatically wrong.

Why attribution needs axioms, not intuition

The naive way to explain a prediction is to read the raw gradient $\partial f / \partial x$: the sensitivity of the output to each input sample. It is fast and it is misleading, because a trained network saturates. Once a feature is large enough to have already flipped the decision, its local gradient is near zero, so the very sample that caused the alert receives no credit. Attribution methods that earn trust are the ones defined by properties they guarantee rather than by a heuristic they happen to compute.

Two axioms carry most of the weight. Sensitivity: if changing one coordinate changes the prediction, that coordinate gets nonzero attribution. Completeness (also called efficiency in the Shapley literature): the attributions add up exactly to the difference between the prediction and a reference prediction. Completeness is what makes an attribution map auditable, because you can sum the highlighted region and check that it accounts for the score you are trying to explain, rather than admiring a pretty heatmap that sums to nothing in particular.

Key Insight

Every attribution method for sensor data is silently answering the question "contribution relative to what?" That reference is the baseline (Integrated Gradients) or the background distribution (SHAP). For images the field settled on a black image as "absence", and the habit carried over as a zero-vector baseline for time series. For a sensor stream that habit is a bug. Zero is not the absence of an accelerometer reading, it is free-fall; zero is not the absence of an ECG, it is asystole; zero is not silence on a microphone unless the signal was already zero-mean. The baseline encodes what your explanation treats as the uninformative null state, and if the null state is itself a dramatic physical event, completeness will faithfully attribute the prediction to the gap between "normal" and "catastrophe" instead of to the feature you cared about.

Integrated Gradients: crediting the whole path

Integrated Gradients (Sundararajan, Taly, and Yan, 2017) fixes gradient saturation by not trusting the gradient at a single point. It defines a straight-line path from a baseline $x'$ to the input $x$ and accumulates the gradient along it. For coordinate $i$,

$$ \mathrm{IG}_i(x) = (x_i - x'_i)\int_{0}^{1} \frac{\partial f\big(x' + \alpha\,(x - x')\big)}{\partial x_i}\, d\alpha. $$

The $(x_i - x'_i)$ factor is the displacement of that coordinate from baseline to input; the integral averages its sensitivity over every interpolation along the way, so a feature that mattered early on the path still earns credit even after it saturates. In practice the integral is a Riemann sum over 20 to 300 interpolation steps, and the whole computation is a batched forward-backward pass, cheap enough to run on every flagged window in a fleet. The payoff is the completeness identity: summing $\mathrm{IG}_i$ over all coordinates equals $f(x) - f(x')$ exactly (up to the Riemann approximation), which gives you a built-in convergence check. If the sum drifts from that target, add steps.

For a multivariate window the attribution has the same $C \times T$ shape as the input, so you can render it as a heatmap over channels and time, or marginalize it: sum over time to rank which sensor drove the decision, or sum over channels to find which instant. The baseline choice reappears here with teeth. A defensible default for a mean-removed sensor window is the per-channel training mean (a "typical quiet" state), or, better, an average over several plausible baselines drawn from healthy operation, which is exactly the expectation that SHAP formalizes next.

import torch

def integrated_gradients(model, x, baseline, target, steps=64):
    """IG for one sensor window. x, baseline: (C, T) tensors. target: output index.
    Returns a (C, T) attribution map whose sum approximates f(x) - f(baseline)."""
    alphas = torch.linspace(0, 1, steps).view(-1, 1, 1)          # (steps, 1, 1)
    path = baseline.unsqueeze(0) + alphas * (x - baseline).unsqueeze(0)
    path.requires_grad_(True)
    out = model(path)[:, target]                                 # (steps,)
    grads = torch.autograd.grad(out.sum(), path)[0]              # (steps, C, T)
    avg_grad = grads.mean(dim=0)                                 # trapezoid-ish mean
    attributions = (x - baseline) * avg_grad                    # (C, T)
    return attributions

# convergence sanity check: does completeness hold?
# attributions.sum()  ~=  model(x)[target] - model(baseline)[target]

A self-contained Integrated Gradients pass over a multivariate sensor window. The final comment is the completeness check: if attributions.sum() does not track the prediction gap, the Riemann sum has too few steps. The baseline argument is where the physics lives, not a throwaway zero.

The code above is deliberately bare so the mechanism is visible. In production you would not hand-roll the Riemann sum, the gradient bookkeeping, or the multi-baseline averaging.

Library Shortcut

Captum's IntegratedGradients replaces the whole function above, plus multi-baseline expectation, batching, and the convergence-delta report, with three lines: ig = IntegratedGradients(model); attr, delta = ig.attribute(x, baselines=bg, target=y, n_steps=128, return_convergence_delta=True). That is roughly 15 lines of hand-written path, gradient, and check logic collapsed to 3, and Captum returns the completeness gap (delta) for free so you never ship an under-converged map. Captum also supplies GradientShap and DeepLiftShap for the Shapley variants below, all with the same call shape. We return to Captum as a review surface in Section 67.6.

SHAP for time series: the same axioms, from game theory

SHAP (SHapley Additive exPlanations) reaches attribution from cooperative game theory. Treat each input coordinate as a player and $f$ as the payoff. The Shapley value of coordinate $i$ is its average marginal contribution over every ordering in which coordinates are added to the "coalition", with absent coordinates replaced by samples from a background distribution:

$$ \phi_i = \sum_{S \subseteq N \setminus \{i\}} \frac{|S|!\,(|N|-|S|-1)!}{|N|!}\,\big[\,f(S \cup \{i\}) - f(S)\,\big]. $$

Shapley values are the unique attribution satisfying local accuracy (their completeness), missingness, and consistency, which is why SHAP and Integrated Gradients so often agree: IG is, under mild conditions, the Aumann-Shapley value of the same game, computed by a path integral instead of a sum over coalitions. The exact sum is exponential in the number of players, so a sensor window of $C \times T$ samples is hopeless at full resolution. This is where SHAP-TS becomes an engineering discipline rather than an off-the-shelf call: you must choose the players, the background, and the approximator.

Players (granularity). A single sensor sample is rarely a meaningful unit; adjacent samples are correlated and no operator reasons "millisecond 4,113 caused the fault". Group the window into segments (fixed strides, or change-point blocks, or per-channel bands) so that a player is a physically interpretable chunk. Fewer players also makes the approximation tractable.
Background. "Absent" means resampled from a background set of normal-operation windows, so the explanation reads as "relative to healthy behavior" rather than "relative to zero". This is the SHAP-native cure for the baseline trap.
Approximator. KernelSHAP (model-agnostic, needs many forward passes), DeepSHAP / GradientSHAP (gradient-based, fast, for differentiable nets), or TreeSHAP if you explain a gradient-boosted model over engineered features from Chapter 8.

Field story: a wearable fall-alert that blamed the wrong second

A wearables team ships a fall detector on a wrist accelerometer and gyroscope, a six-channel window at 50 Hz over four seconds, feeding a small temporal-convolution net from Chapter 14. A clinician reviewing false alarms asks the obvious question: why did it fire? The first attempt uses Integrated Gradients with a zero baseline. The heatmap lights up the pre-impact quiet segment, and everyone is confused, because "the person standing still" is apparently the cause of the fall. The bug is the baseline: against a zero vector, the near-1 g gravity component of the resting hand is a large displacement, so completeness dutifully credits gravity. Switching to a background of the wearer's own quiet-standing windows (SHAP with a personalized background, and IG re-run with the same per-channel mean baseline) moves the attribution onto the 300 ms deceleration spike and the post-impact stillness, which is what a clinician would circle by hand. Same model, same window, same axioms: only the reference changed, and the explanation went from nonsense to actionable. The team folds this into the alert UI so each fall event ships with its attribution ribbon, and false alarms are triaged in seconds. It also connects to Chapter 37-style anomaly review, where the same "relative to normal" framing is the whole point.

Reading a temporal attribution without fooling yourself

Two disciplines keep attribution maps from becoming decorative. First, always report the completeness residual. Both IG (via its convergence delta) and SHAP (via local accuracy) tell you how much of the prediction the map failed to explain; a large residual means the map is incomplete and the top-ranked feature may be an artifact of under-convergence, not a real driver. Second, attribution is per-instance, not a global feature-importance. A map explains this window; aggregating many maps to claim "channel 3 is important overall" is a different, weaker statement and is easy to overstate. When a model is well calibrated (the machinery of Chapter 18), the attribution of a high-confidence prediction is worth more scrutiny than one drawn from a coin-flip output near the decision boundary, where the gradient landscape is noisy and the map is unstable.

Attribution answers "which input samples", not "why in mechanism" and not "what would have changed the outcome". The counterfactual question is the subject of Section 67.3, and the specifically temporal failure modes of saliency, alignment jitter, and adjacent-sample smearing get their own treatment next.

Research Frontier

Baseline selection for time series is an open problem, not a solved default. Recent work replaces the single baseline with learned or distributional references and adapts Integrated Gradients to the sequential structure directly: Temporal Integrated Gradients and WinIT integrate over time-aware paths so that credit respects temporal dependence rather than treating each timestep as an independent coordinate. The Dynamask line learns a smooth per-timestep saliency mask under perturbation, and the toolkit landscape is consolidating in TSInterpret and Captum (both revisited in Section 67.6), with faithfulness measured on XTSC-Bench in Section 67.7. The practical takeaway for now: prefer a distribution of physically plausible baselines over any single fixed one, and validate the map against a faithfulness metric rather than eyeballing it.

Exercise

Take a trained binary anomaly model on a three-channel vibration window (say 128 timesteps). (1) Compute Integrated Gradients twice, once with a zero baseline and once with the per-channel training mean, and plot the completeness delta for each; report which converges tighter at 64 steps. (2) Marginalize each attribution map over time to rank the three channels, and over channels to find the peak instant. (3) Now run KernelSHAP with a background of 50 normal windows, grouping the input into 16 time segments per channel, and compare its channel ranking to IG's. Where they disagree, decide which reference you trust and justify it in one sentence.

Self-Check

State the completeness axiom and explain why a zero baseline can make a physically correct model produce a nonsensical attribution map on an accelerometer window.
Why does the raw gradient $\partial f/\partial x$ systematically under-credit the very feature that triggered a saturated decision, and how does the path integral in Integrated Gradients repair this?
You have a differentiable transformer and need attributions on thousands of flagged windows per hour. Would you reach for exact KernelSHAP or a gradient-based method (IG / GradientSHAP), and what is the tradeoff you are accepting?

What's Next

In Section 67.2, we zoom in on the specifically temporal hazards of these maps: why saliency over time is fragile to small phase shifts and resampling, why adjacent-timestep smearing fakes precision the model never had, and how to tell a genuine temporal driver from an artifact of the attribution method itself.