"They told me the fused model scored ninety-four. I asked, better than which sensor alone? The silence that followed was the most informative measurement in the room."
A Skeptical AI Agent
Prerequisites
This section assumes the fusion architectures of Section 48.2, the missing-modality behavior of Section 48.4, and the confidence weighting of Section 48.5. The leakage-safe splitting discipline it leans on is built from the ground up in Chapter 5 and revisited as a full protocol in Chapter 65. Calibration terminology comes from Chapter 18.
The Big Picture
A fused system is easy to over-praise. Stack three sensors, report one accuracy number, and it will almost always look impressive. The number is nearly meaningless on its own, because it hides the only questions that matter: did fusion actually help, where did it help, and does it still help when a sensor drops out or drifts. Evaluating a fused system is not measuring one score; it is measuring a gain against honest baselines, across the conditions each sensor was supposed to cover, with the failure modes deliberately switched on. Skip that discipline and you ship a system that is expensive, complex, and no better than its cheapest single sensor.
Fusion must beat its own best sensor, not zero
The first and most violated rule of fused evaluation is the choice of baseline. A fused model that scores \(0.94\) is only interesting relative to what each modality achieves alone. Define the fusion uplift as the gap between the fused score and the best single-modality score on the identical test set:
\[ \Delta_{\text{fusion}} = M_{\text{fused}} - \max_i M_i, \]where \(M_i\) is the metric for a model trained and evaluated on modality \(i\) only. What this measures is whether the machinery of fusion earned its complexity. Why it matters: a great many published and deployed "fusion wins" evaporate when someone finally trains a competent single-sensor baseline and discovers the camera alone already scored \(0.93\). How to compute it: hold the split, preprocessing, and evaluation code fixed, swap only the input, and re-run. When \(\Delta_{\text{fusion}}\) is near zero or negative, the honest report is that fusion did not help here, and the extra sensor is a cost with no return.
Two further baselines sharpen the picture. A naive-combination baseline (averaging or voting the single-sensor predictions with no learned interaction) tells you whether your fancy cross-attention fusion beats a one-line ensemble. An oracle baseline, which picks the correct answer whenever any single modality was right, upper-bounds what perfect fusion could extract from these sensors; the distance between your model and the oracle is the headroom still on the table. Report all three and the single accuracy number stops being able to hide.
Key Insight
The metric of a fused system is a difference, not a level. Any evaluation that reports one absolute score without at least the best single-modality baseline computed on the same split, same preprocessing, same seed, is not an evaluation of fusion; it is an evaluation of the dataset's difficulty. Co-compute every baseline in one pass over one configuration so the subtraction \(M_{\text{fused}} - \max_i M_i\) is actually valid.
Ablation and degradation curves: measure the graceful part
A single test number assumes all sensors are present and healthy, which is exactly the assumption the field breaks. The purpose of fusion, as Section 48.4 argued, is partly to survive the loss of a modality. So the evaluation must survive it too. Two tools do this work. A leave-one-modality-out ablation re-runs the fused model with one input zeroed or masked and reports the resulting score; the drop tells you how much that sensor was carrying and whether the model leaned on it dangerously hard. A degradation curve sweeps a corruption (added noise, dropout probability, latency, occlusion) along the x-axis and plots the fused metric against it, ideally overlaid with the single-sensor curves.
The shape of that curve is the deliverable. A well-designed fused system degrades gracefully: its curve stays above every single-sensor curve as conditions worsen, and it flattens rather than falling off a cliff when one modality becomes useless. A brittle system shows the opposite signature, a fused curve that tracks its most-trusted sensor and collapses the moment that sensor is corrupted, revealing that the model never truly learned to reweight. Figure 48.6.1 sketches these two signatures side by side.
Figure 48.6.1. Left: a graceful fused curve stays above both single-sensor curves and flattens as noise rises. Right: a brittle fused curve hugs its dominant sensor A and collapses with it, the tell that the model never learned to reweight onto sensor B.
The code below computes an uplift number and a leave-one-out ablation from stored per-modality and fused predictions. It is the smallest honest evaluation of a two-sensor fusion, and Listing 48.6.1 is what you run before believing any fusion claim.
import numpy as np
def accuracy(pred, y):
return float((pred == y).mean())
# y: true labels; probs_* : predicted class probabilities (N, C)
def evaluate_fusion(y, probs_A, probs_B, probs_fused):
accA = accuracy(probs_A.argmax(1), y)
accB = accuracy(probs_B.argmax(1), y)
accF = accuracy(probs_fused.argmax(1), y)
# ablation: fuse with a modality masked to a uniform (uninformative) prior
C = probs_A.shape[1]
flat = np.full_like(probs_A, 1.0 / C)
accF_noA = accuracy((flat * probs_B).argmax(1), y) # sensor A dropped
accF_noB = accuracy((probs_A * flat).argmax(1), y) # sensor B dropped
# oracle: correct if EITHER single sensor was correct
oracle = accuracy(np.where(probs_A.argmax(1) == y, y, probs_B.argmax(1)), y)
return {
"best_single": max(accA, accB),
"fused": accF,
"uplift": accF - max(accA, accB),
"drop_if_no_A": accF - accF_noA,
"drop_if_no_B": accF - accF_noB,
"oracle_ceiling": oracle,
}
The Right Tool
Turning that dictionary of numbers into a full stratified report (per-slice accuracy, per-modality ablation, and confidence intervals across operating conditions) is where a library saves real effort. Hand-rolling the grouping, aggregation, and bootstrap resampling runs roughly 40 to 60 lines of index-juggling that is easy to get wrong at the group edges. A pandas groupby plus a resampling helper collapses it:
import pandas as pd
# df has columns: correct (0/1), slice (e.g. "night", "indoor"), model
report = (df.groupby(["model", "slice"])["correct"]
.agg(["mean", "count"])
.rename(columns={"mean": "accuracy"}))pandas aggregation replaces roughly 50 lines of manual per-slice bookkeeping, producing accuracy and support for every model-by-condition cell. It does the grouping, not the judgment about which slices matter.The library handles the bookkeeping. Choosing which slices to stratify on, the subject of the next subsection, remains your design decision.
Stratify by the regimes each sensor was chosen to cover
An aggregate score averages over exactly the conditions where fusion is supposed to prove itself. Complementary sensors were chosen because each dominates a different regime: the camera in daylight, the radar in fog, the IMU during brief visual dropout. If your test set is 90% easy daylight, an aggregate number rewards the camera and drowns out the radar's contribution, precisely the regime where fusion earns its keep. The fix is stratified evaluation: partition the test set into the operating regimes that motivated each modality, and report uplift within each stratum. What you are checking is that fusion helps most where its weakest single sensor is, not just on average. This is the same worst-slice thinking that Chapter 66 formalizes for distribution shift; here it is the difference between a fused system that is genuinely robust and one that merely rides an easy majority slice.
In Practice: automotive perception at dusk
A driver-assistance team fuses camera and radar for forward-object detection and reports \(0.96\) average precision, a healthy \(+0.03\) over the camera alone. The uplift looks modest until they stratify by lighting. In daytime, camera-only already reaches \(0.97\) and fusion adds almost nothing. At dusk and in low sun glare, camera-only falls to \(0.71\) while the fused system holds at \(0.90\), because the radar range and closing-speed cues survive the glare that blinds the camera. The aggregate number had averaged a near-zero daytime uplift with a decisive nineteen-point night-time uplift into a bland \(+0.03\). The stratified report is what justified keeping the radar in the bill of materials, and it is what told the safety case exactly which conditions the second sensor was protecting.
Calibration and the leakage traps unique to fusion
Accuracy is not the whole story for a fused system, because fusion combines evidence and a miscalibrated combiner combines it wrongly. As Section 48.5 showed, confidence weighting only works if each modality's confidence means what it claims. So evaluate the fused system's calibration, not only its accuracy: an over-confident fused model, one whose \(0.9\)-probability predictions are right only \(0.7\) of the time, will mislead every downstream decision that trusts its uncertainty. The expected calibration error bins predictions by confidence and measures the average gap between confidence and observed accuracy:
\[ \mathrm{ECE} = \sum_{b=1}^{B} \frac{|S_b|}{N}\,\bigl|\,\mathrm{acc}(S_b) - \mathrm{conf}(S_b)\,\bigr|, \]where \(S_b\) is the set of predictions in confidence bin \(b\). Report ECE for the fused model alongside its accuracy, and re-check it under the same degradation sweeps, because a model can stay accurate while becoming badly over-confident as a sensor drops out. The calibration machinery to fix a bad ECE is developed in Chapter 18.
Common Trap: cross-modal leakage inflates fused scores
Fused datasets carry a leakage risk that single-sensor datasets do not. When two modalities are recorded together, a nuisance variable shared by both (the same recording session, the same device, the same subject wearing both sensors) can let a model identify the sample rather than solve the task, and fusion amplifies this by giving the model two correlated shortcuts instead of one. If your split puts the same subject or session on both sides of the train/test line, the fused score is inflated and the uplift is partly memorized correlation. Split by the coarsest shared unit (subject, then session, then device), exactly as Chapter 5 prescribes, and re-measure uplift under that grouped split before trusting it.
Exercise
Take any two-modality dataset you can access (or simulate two noisy views of one label). Train three models: modality A alone, modality B alone, and a simple late-fusion average. Compute the fusion uplift, the two leave-one-out drops, and the oracle ceiling with Listing 48.6.1. Then build one degradation curve by adding increasing Gaussian noise to modality A only, and plot the fused, A-only, and B-only accuracies against noise level. State in one sentence whether your fused curve is graceful or brittle, and justify it from the plot.
Self-Check
1. Why is a single fused accuracy number nearly uninformative, and what is the minimum baseline that makes it meaningful?
2. What does a degradation curve that hugs its most-trusted single-sensor curve tell you about the fusion model, and why is that dangerous?
3. How can recording two modalities from the same subject inflate a fused system's measured uplift, and which splitting rule defends against it?
What's Next
In Section 48.7, we turn the evaluation lessons of this section into a catalogue of concrete design patterns and anti-patterns: the recurring shapes of fusion that succeed, and the seductive ones (the dominant-sensor crutch, the untested missing-modality path, the leaked correlated split) that this section's measurements are built to catch before they ship.