Concept- and prototype-based explanations

"Attribution tells me which millisecond mattered. A prototype tells me what I was reminded of. Only one of those survives a conversation with a technician."

A Case-Based AI Agent

This section assumes you are comfortable with learned embeddings for sensor windows (Chapter 13), with contrastive representation learning that shapes those embedding spaces (Chapter 17), and with the shapelet and feature ideas from Chapter 8. Prototypes are, in effect, learnable shapelets living in embedding space.

Prototype networks: reasoning by resemblance

The prototype recipe adapts the vision idea of "this looks like that" (ProtoPNet, Chen et al. 2019) to sequences. An encoder \(f\) maps a window \(x\) to an embedding \(z = f(x) \in \mathbb{R}^d\). A prototype layer holds \(K\) learnable vectors \(p_1,\dots,p_K\), each tied to a class. The network scores a window by its similarity to every prototype, then classifies from those similarities alone:

Because the final decision is a sparse weighted sum over similarities, every prediction decomposes into "activated 0.81 on prototype #12, which is the outer-race defect from bearing B-207, logged 2024-03." The crucial trick, from ProSeNet (Ming et al. 2019) and its descendants, is a projection step: after training, each prototype vector is snapped onto the embedding of a real training window nearest to it. That guarantees the explanation is an actual measured signal you can plot, not a synthetic point in latent space that corresponds to nothing physical.

Three loss terms make the prototypes usable. A clustering term pulls every training embedding close to some prototype of its class; a separation term pushes embeddings away from other classes' prototypes; and a diversity term penalizes prototypes that collapse onto each other, so you get distinct failure archetypes rather than twelve copies of one. Getting the similarity metric right matters: Euclidean distance in a poorly aligned embedding treats two identical shapes offset by three samples as different. Encoders trained with temporal augmentation, or a soft dynamic-time-warping distance, restore the shift invariance that raw \(\ell_2\) throws away.

Concept-based explanations: naming the axes

Prototypes explain by example; concepts explain by attribute. Testing with Concept Activation Vectors (TCAV, Kim et al. 2018) asks a different question: how sensitive is the model's fault prediction to a human concept such as "high-frequency ringing"? You assemble a small set of windows that exhibit the concept and a set that does not, train a linear probe in an intermediate activation layer to separate them, and take the probe's normal vector \(v_C\) as the concept direction. The concept's influence on class \(c\) is the fraction of examples whose directional derivative along \(v_C\) is positive:

A score near 1 means the concept consistently pushes the model toward class \(c\); near 0.5 means it is irrelevant. This is powerful for auditing: you can test whether a sleep-stage model relies on "delta-wave power" (a clinically valid concept) or on "electrode pop artifacts" (a leakage concept it should ignore). Concept Bottleneck Models (Koh et al. 2020) go further and force the architecture through a named concept layer: the network first predicts interpretable concepts, then predicts the label only from those concepts. That makes the model steerable, because a technician who corrects a wrong concept value gets a corrected downstream prediction, but it costs accuracy whenever the pre-declared concepts miss something the raw signal contained.

Building a prototype layer on sensor embeddings

The code below defines a minimal prototype head over any frozen or trainable encoder, and shows the two operations that make it interpretable: scoring by similarity, and the post-training projection of each prototype onto its nearest real embedding. It is deliberately small so the mechanism is visible; production trees add the clustering, separation, and diversity losses discussed above.

import torch
import torch.nn as nn
import torch.nn.functional as F

class PrototypeHead(nn.Module):
    def __init__(self, dim, n_prototypes, n_classes, tau=1.0):
        super().__init__()
        self.prototypes = nn.Parameter(torch.randn(n_prototypes, dim))
        self.classifier = nn.Linear(n_prototypes, n_classes, bias=False)
        self.tau = tau

    def similarities(self, z):                       # z: (B, dim) embeddings
        d2 = torch.cdist(z, self.prototypes) ** 2    # squared L2 to each prototype
        return torch.exp(-d2 / self.tau)             # (B, n_prototypes), in (0, 1]

    def forward(self, z):
        s = self.similarities(z)
        return self.classifier(s), s                 # logits AND the explanation vector

    @torch.no_grad()
    def project_to_data(self, bank_z):               # bank_z: (N, dim) real windows
        d2 = torch.cdist(self.prototypes, bank_z) ** 2
        nearest = d2.argmin(dim=1)                    # index of closest real window
        self.prototypes.copy_(bank_z[nearest])       # snap prototypes onto real data
        return nearest                                # rows to plot as the "that"

# usage: each prediction carries s, and nearest[k] names the plottable exemplar
head = PrototypeHead(dim=128, n_prototypes=20, n_classes=4)
z = torch.randn(8, 128)                              # encoder output for 8 windows
logits, s = head(z)
top_proto = s.argmax(dim=1)                          # which archetype each window matched

Notice that forward returns s alongside the logits: the explanation is not computed by a separate tool afterward, it falls out of the forward pass. That is the structural advantage of self-explaining models over the post-hoc attribution of Section 67.1.

When each family earns its keep

Prototypes shine when the operator's mental model is case-based: technicians already reason "this sounds like the pump we replaced last spring," so a nearest-archetype explanation lands without training. They degrade when classes have no compact archetype (diffuse, high-variance faults need dozens of prototypes and the explanation stops being concise). Concepts shine when you have a pre-existing vocabulary to audit against, especially for detecting spurious cues and satisfying a reviewer who asks "does it use anything it should not?" They degrade when the true discriminative feature has no name yet, because a concept bottleneck can only reason with concepts you thought to declare. In predictive-maintenance workflows (Chapter 36) the two compose well: prototypes for the front-line alert, TCAV as a periodic audit that the prototype library has not drifted onto sensor artifacts. Always fix prototypes and concept sets on the training split only, using the leakage-safe partitions of Chapter 65; a prototype projected onto a test window has quietly memorized the test set.

What's Next

In Section 67.5, we move from "what does this window resemble" to "what caused it": root-cause graphs turn a fired alert into a directed structure over sensors, subsystems, and events, so an explanation stops being a single archetype and becomes a traceable chain through the physical system.