Interpretability

"An accurate forecast a control-room operator will not act on is a research demo. Show me which three sensors and which two hours drove the alarm, and now it is a tool."

An Accountable AI Agent

What there is to explain, and why it is different here

An image classifier explains itself with a saliency map over pixels; a sequence model highlights time steps. A sensor GNN has three attribution surfaces at once, and a useful explanation names all three. First, node importance: which sensors' current features drove this output. Second, edge importance: which relationships carried the decisive messages, since the model's whole premise is that information flows along edges. Third, temporal and feature importance: for the spatiotemporal models of Section 54.3, which time steps and which input channels mattered. The why this matters is not abstract. In a regulated setting (grid operations, water safety, clinical monitoring) an unexplained automated decision is often not permitted; Chapter 70 takes up the legal side. Even without a regulator, an explanation is how you debug the graph: if the model attributes a burst-detection to an edge that does not correspond to a real pipe, you have caught either a bad topology or a spurious correlation before it fails in production.

Three families of method

The how splits into three approaches with different costs and different failure modes. Attention weights are the cheapest: a graph attention network (GAT) already computes a learned coefficient \(\alpha_{ij}\) for each edge, and reading them off looks like an explanation for free. It sometimes is, but attention is a soft routing weight, not a guaranteed attribution; the same output can often be produced by very different attention patterns, so treat attention as a hypothesis to be verified, never as a certificate. Gradient and perturbation attribution ports the tools of Chapter 67 to the graph: integrated gradients over node features, or occlusion, where you zero a node or cut an edge and measure how much the prediction moves. Learned explanation masks are the GNN-specific frontier. GNNExplainer optimizes a continuous mask \(\mathbf{M} \in [0,1]^{|E|}\) over edges (and optionally over features) to find the smallest subgraph that preserves the model's prediction, by maximizing the mutual information between the masked-graph output and the original,

where the \(\ell_1\) penalty forces sparsity so the answer is a handful of edges a human can read, not a diffuse cloud. PGExplainer trains a small network to predict such masks, so once trained it explains new instances in a single forward pass instead of a per-instance optimization, which is what you need to explain a live fleet.

import torch
from torch_geometric.explain import Explainer, GNNExplainer

# `model` is a trained GNN; `data` holds x, edge_index for one sensor-network snapshot.
explainer = Explainer(
    model=model,
    algorithm=GNNExplainer(epochs=200, lr=0.01),
    explanation_type="model",              # explain what the model does, not the label
    node_mask_type="attributes",           # which input features matter
    edge_mask_type="object",               # which edges (relationships) matter
    model_config=dict(mode="regression", task_level="node", return_type="raw"),
)

expl = explainer(data.x, data.edge_index, index=14)   # explain the alarm at node 14
edge_score = expl.edge_mask                            # one importance in [0,1] per edge
top = edge_score.topk(3).indices                       # the 3 decisive relationships
print("decisive edges:", data.edge_index[:, top].t().tolist())
print("feature importance:", expl.node_mask[14].round(decimals=2).tolist())

Listing 54.7 makes the abstract mask concrete: the output is not a scalar confidence but the three edges and the feature channels the model actually used, expressed as objects an engineer can trace on a schematic.

Is the explanation faithful? Measuring, not trusting

An explanation that looks plausible can still be unfaithful, describing what a human expects rather than what the model computed. So you measure faithfulness with a perturbation protocol. Fidelity-minus asks how much the prediction degrades when you keep only the explanatory subgraph and drop the rest; a faithful explanation loses little. Fidelity-plus asks how much it degrades when you remove exactly the explanatory subgraph and keep everything else; a faithful, decisive explanation loses a lot. Sparsity counts the edges retained, because an "explanation" that keeps half the graph explains nothing. A good method pushes fidelity-plus high and the retained set small at once. Two cautions carry over from the rest of the book. First, attention magnitude is not attribution: a heavily attended edge that, when cut, barely moves the output was not actually decisive, and only the perturbation test reveals that. Second, evaluate explanations on held-out events, never on the events the mask was fit to, or you measure memorization, exactly the leakage failure mode of Chapter 5.

Reading the learned graph itself

A distinct opportunity, unique to the adaptive models of Section 54.3, is that some GNNs learn their adjacency. Graph WaveNet's self-adaptive adjacency matrix is, after training, a dense map of which sensors the model decided are coupled, learned from data rather than handed a topology. Inspecting it is a form of global interpretability: strong learned edges that match the known physical network are reassuring, and strong edges that do not are hypotheses about hidden couplings (a shared upstream disturbance, an unmodeled electrical path) worth investigating. The same caution from structure learning applies: a learned edge that improves prediction need not be a causal wire, since many adjacencies explain the same data equally well. Read the learned graph as a source of testable hypotheses, then confirm the important ones with an intervention or a perturbation test, not as a discovered ground truth.

What's Next

In Chapter 55, the falsifiable explanations of this section meet their natural ally: physics. Digital twins and physics-informed models give the sensor network a simulated ground truth to check learned couplings against, and constrain the model so its explanations respect conservation laws by construction rather than by luck.