Human-review tooling (TSInterpret, Captum)

"An explanation that lives only in a notebook is a private opinion. An explanation a tired technician can accept or overrule at 3 a.m. is a control."

A Well-Tooled AI Agent

This section assumes you can already produce the attributions and prototypes of Sections 67.1 through 67.5, and that you understand the model-monitoring context in which review queues live (Chapter 69). Because reviewers act on flagged windows, you also need the anomaly-scoring backbone that produces the flags (Chapter 37) and the calibrated confidence that decides which flags reach a person at all (Chapter 18).

Why a tooling layer, not another method

A production sensor model rarely commits to one explainer. You want gradient-based attribution for speed on every alert, occlusion for a slower but model-agnostic second opinion, and a counterfactual when the reviewer asks "what would have made this normal?" Wiring three research repos together by hand means three tensor conventions, three normalization schemes, and three subtly different notions of what the "baseline" is. TSInterpret (Höllig, Kulbach, and Thoma, 2022) exists to remove that friction: it presents feature-attribution methods (Integrated Gradients, GradCAM, LEFTIST, TSR) and instance-based methods (counterfactuals, Native-Guide) behind one explain() call that always takes a window shaped \((\text{samples}, \text{channels}, \text{time})\) and always returns something plottable. The value is not a new algorithm; it is that every explainer becomes swappable, so a reviewer's dashboard shows the same layout whether the number underneath came from a gradient or an occlusion sweep.

Captum (Kokhlikyan et al., 2020) sits one level below as the differentiable attribution library for any PyTorch model. It supplies the mathematics: IntegratedGradients, DeepLift, GradientShap, Occlusion, and the captum.concept TCAV machinery used in Section 67.4. For a 1D sensor CNN it works out of the box because a window is just a tensor; the only care needed is choosing a physically meaningful baseline (a zeroed window implies "silence," which is wrong for a sensor with a nonzero DC bias). The division of labor is clean: Captum computes, TSInterpret adapts and standardizes, and your review UI consumes a uniform result object.

From attribution tensor to a review loop

A raw attribution is a tensor the same shape as the window. A review case is that tensor plus everything the human needs to rule on it: the input window, the model's prediction and calibrated confidence, the top attributed channels and time spans, one nearest historical prototype (67.4), and a field for the reviewer's verdict. The loop has four stages. Triage orders the queue, usually by a mix of predicted severity and low confidence, so scarce attention lands where a human is most likely to change the outcome. Presentation renders the case with the attribution overlaid on the signal, not as a separate heatmap the eye has to align by hand. Adjudication captures a structured verdict (confirm, override, escalate) plus a free-text reason. Feedback writes the verdict back as a label, closing the loop with the retraining triggers of Chapter 69. Skipping the feedback stage is the most common failure: teams ship a beautiful dashboard, collect thousands of expert verdicts, and throw them away.

The code below builds one review case for a sensor-window classifier using Captum's Integrated Gradients, then reduces the full attribution to the compact summary a reviewer reads first. It is the exact bridge between the method and the UI.

import torch
from captum.attr import IntegratedGradients

def build_review_case(model, window, channel_names, baseline=None):
    model.eval()
    window = window.unsqueeze(0)                     # (1, channels, time)
    logits = model(window)
    prob, pred = torch.softmax(logits, dim=1).max(dim=1)

    if baseline is None:                             # per-channel mean, not zeros
        baseline = window.mean(dim=2, keepdim=True).expand_as(window)

    ig = IntegratedGradients(model)
    attr = ig.attribute(window, baselines=baseline, target=pred, n_steps=64)
    attr = attr.squeeze(0)                           # (channels, time)

    per_channel = attr.abs().sum(dim=1)              # which sensor mattered
    peak_time   = attr.abs().sum(dim=0).argmax()     # which instant mattered
    top_ch      = per_channel.argmax()

    return {
        "prediction":  pred.item(),
        "confidence":  round(prob.item(), 3),        # gate triage on this
        "top_channel": channel_names[top_ch],
        "peak_time":   int(peak_time),
        "attribution": attr,                         # full map for the overlay plot
    }

# usage: one call per flagged window feeds one card in the review queue
case = build_review_case(model, flagged_window, ["ax", "ay", "az", "gx", "gy", "gz"])

The summary fields matter as much as the tensor. A reviewer scanning a queue reads "IMU alert, driven by az at t=190" and triages in a second; only on the ambiguous cases do they open the overlay. That two-tier presentation, one-line summary then full plot on demand, is what keeps a human review loop fast enough to run on a real fleet.

What a reviewer actually needs on screen

A usable review card shows five things in a fixed layout: the raw signal for the driving channels, the attribution overlaid as color or shading directly on that signal, the prediction with its calibrated confidence, one nearest historical case for context, and a compact verdict control. Three rules earn their place. First, overlay, do not juxtapose: a heatmap drawn beside the signal forces the reviewer to align two x-axes by eye, and Section 67.2 already showed how easily temporal saliency misleads even when correctly aligned. Second, show confidence honestly: an uncalibrated 0.99 invites rubber-stamping, so pipe the conformal or temperature-scaled score from Chapter 18, not the raw softmax. Third, make disagreement cheap: a one-click override with a required reason produces the highest-value training labels you will ever collect, because they mark exactly where the model and an expert diverge.

Wiring review into operations

The review loop is not a standalone app; it is a stage in the fleet pipeline. Alerts arrive from the monitoring layer (Chapter 69), the tooling attaches an explanation and a triage score, a human adjudicates, and the verdict flows back as a label and as an audit record. Two practices keep it honest. Cache the explanation with the alert, because recomputing an attribution days later against a since-updated model yields a different picture than the reviewer saw, and your audit trail must show what was actually on screen. And always draw the prototypes and baselines used in the tooling from the training split only, using the leakage-safe partitions of Chapter 65; a "nearest historical case" pulled from the test set has quietly leaked evaluation data into the operator's judgment.

What's Next

In Section 67.7, we ask the question this whole chapter has deferred: is any of it faithful? Explanation-faithfulness benchmarks such as XTSC-Bench give a principled way to score whether an attribution reflects the model's real reasoning or just tells a plausible story, so you can choose the explainer your review tooling should trust.