Fusion interpretability

"A fusion model with four sensors and one opinion is not fusing. It is delegating, and lying to you about it."

An Auditing AI Agent

This is an advanced, research-frontier section. It assumes the cross-attention machinery of Section 50.2, the gating of Section 50.3, and the calibration view of Chapter 18. General single-stream attribution (saliency, integrated gradients, root-cause analysis) lives in Chapter 67; here we specialize those ideas to the one axis that is unique to fusion, the modality.

Two questions, two granularities

Fusion interpretability answers questions at two levels, and confusing them is the most common mistake. The modality level asks how much each sensor contributed to a prediction: did the collision warning come from radar, camera, or their interaction? The token level asks which parts of a modality mattered: which radar bin, which image patch, which time window. Token-level explanation is ordinary saliency applied per stream and is covered in Chapter 67. The modality level is new, and it is the one that governs safety and graceful degradation, because a model that scores its whole decision on one sensor has no fallback when that sensor fails, no matter how good its token-level maps look.

The clean way to define modality contribution borrows the Shapley value from cooperative game theory. Treat the \(M\) modalities as players and the model output (a logit or a score) as the payoff \(v(S)\) of a coalition \(S\) of present sensors, where absent sensors are replaced by their missing-modality substitute (the mask or learned null token from Section 50.4). Modality \(i\)'s contribution is its average marginal value across all coalitions:

With \(M\) modalities this is only \(2^M\) forward passes, entirely tractable for the two-to-five sensors typical of real fusion stacks, and it is the principled version of the "remove one sensor and watch accuracy drop" ablation that Lab 50 runs. Shapley values sum exactly to the difference between the full-fusion output and the all-absent baseline, so they give an honest, conservation-respecting decomposition rather than a heuristic ranking.

Measuring per-modality contribution in practice

The practical recipe is a leave-one-modality-out sweep plus, when you can afford it, the full Shapley expansion. For each coalition you replace absent modalities with the same null token the model was trained to fall back on, run a forward pass, and record the target score. The code below computes exact modality Shapley values for a small fusion model and, as a by-product, the leave-one-out drop that most teams report.

import itertools, torch

@torch.no_grad()
def modality_shapley(model, inputs, null, target, M):
    # inputs: list of M modality tensors; null: list of M mask/null substitutes
    def payoff(coalition):                      # coalition: frozenset of present modalities
        x = [inputs[i] if i in coalition else null[i] for i in range(M)]
        return model(x)[target].item()          # a logit or calibrated score
    phi = [0.0] * M
    others = lambda i: [j for j in range(M) if j != i]
    for i in range(M):
        for r in range(M):                      # coalitions of the other modalities, by size
            for S in itertools.combinations(others(i), r):
                S = frozenset(S)
                w = (len(S)*1.0) and 1.0        # placeholder; real weight below
                weight = (torch.tensor(float(len(S)).__index__()).lgamma().exp()
                          * torch.tensor(float(M-len(S)-1)).lgamma().exp()
                          / torch.tensor(float(M)).lgamma().exp()).item()
                phi[i] += weight * (payoff(S | {i}) - payoff(S))
    return phi                                  # sums to v(all) - v(none)

# leave-one-out is the cheap first pass: drop each sensor, watch the score fall
def loo_drop(model, inputs, null, target, M):
    full = model(inputs)[target].item()
    return [full - model([null[i] if j==i else inputs[j]
                          for j in range(M)])[target].item() for i in range(M)]

Run this over a validation set, not one example, and look at the distribution of contributions. A healthy fusion model shows every modality carrying non-trivial average \(\phi_i\) and, more importantly, a contribution that rises for a sensor on the frames where the others are weak. That conditional shift is the fingerprint of genuine fusion; a flat, one-modality-dominant profile is the fingerprint of collapse.

Diagnosing unimodal collapse and shortcuts

Unimodal collapse (also called modality laziness or greedy modality dominance) happens because sensors learn at different speeds. A modality that is easy to fit early, or that happens to correlate with the label through a dataset artifact, gets the optimizer's attention first; the fusion head then down-weights the slower streams and never recovers, since the loss is already low. Three diagnostics catch it. First, the contribution profile above: dominance by one \(\phi_i\) is the direct symptom. Second, per-modality drop-rate curves: sweep each sensor's dropout probability at inference and plot accuracy; a model that barely moves when you delete a sensor was never using it. Third, gate and expert histograms for mixture-of-experts fusion (Section 50.3): if the router sends nearly all mass to experts fed by one modality, the MoE has collapsed to a unimodal model wearing a multimodal costume. These are the same shortcut-learning pathologies that Chapter 68 treats as a safety and spoofing risk: a model that leans entirely on one sensor is a model an attacker only has to fool once.

From explanation to guarantee: calibrating contribution

An attribution number is only useful if it is stable and honest. Two habits make it so. First, report contribution with the uncertainty of the model: a sensor's \(\phi_i\) should be read alongside the model's calibrated confidence from Chapter 18, because a high contribution from an over-confident model is not evidence of good fusion. Second, evaluate attribution leakage-safely: compute contribution profiles on a split whose subjects, sites, and sessions never appear in training, or the profile will simply memorize the same shortcut the accuracy did. A contribution audit that runs on leaked data certifies the leak. Faithfulness itself can be scored: perturb or delete the top-contributing modality and check that the output moves as the attribution predicted (a deletion or insertion curve). If deleting the modality your method calls decisive barely changes the output, the explanation is wrong and should be discarded, not published.

What's Next

In Section 50.7, we turn from understanding the fusion model to shipping it. The interpretability profile you just learned to measure becomes a deployment constraint: a model that leans on an expensive lidar can be pruned toward a cheaper sensor set, but only if you know what each modality actually buys you. We weigh latency, memory, power, and per-sensor cost against the accuracy and robustness each modality contributes, and see how the same contribution numbers guide which sensors to keep, downsample, or drop on the edge.