Missing-modality training and graceful degradation

"They trained me on five senses and deployed me with three. On my first day the fog rolled in and took a fourth, and no one had told me what to do with the silence."

A Suddenly Deaf AI Agent

Why fusion models shatter when a sensor drops out

What goes wrong is specific and mechanical. A fusion head learns a joint decision boundary over the concatenated or attended features of all modalities. Suppose training always presents camera features \(z_c\), radar features \(z_r\), and lidar features \(z_l\). The head learns weights tuned to the statistics of all three arriving together, including their correlations. At inference the lidar fails and you must supply something in place of \(z_l\). The tempting default, feeding a zero vector, is a trap: zero is not "absent," it is a specific, confident input that almost never occurred in training, so the head extrapolates into a region it has never seen. The network does not know the lidar is missing; it reads a vivid, wrong measurement and fuses it with full confidence.

Why this matters more for deep fusion than for the classical Bayesian fusion of Chapter 49 is that a Kalman-style filter has an explicit measurement model: drop a measurement and you simply skip its update step, and the posterior widens in a principled way. A learned fusion head has no such factorization by default. Its reliance on cross-modal correlation, the very thing that makes joint fusion powerful, is exactly what makes it brittle when the correlation partner vanishes. Empirically, a naive tri-modal model fed a zeroed modality often scores below a model trained on the two survivors alone. You paid for a third sensor and got a liability.

Modality dropout: train on the subsets you will face

What modality dropout does is stochastically remove entire modalities during training, the structured analogue of the neuron-level dropout you know from Chapter 13. On each forward pass you sample a subset \(S \subseteq \{c, r, l\}\) of present modalities from some distribution \(p(S)\), mask the rest, and compute the loss only from what remains. Formally, instead of minimizing the full-modality risk, you minimize its expectation over the dropout distribution,

where \(x_S\) keeps only the modalities in \(S\). Why this works: the single set of shared parameters \(\theta\) is now optimized to make a good decision under every subset that \(p(S)\) gives weight to, so the fusion head learns to reweight itself toward whatever evidence is present. The network effectively becomes an ensemble of \(2^M - 1\) sub-models sharing one backbone, one for each non-empty subset of \(M\) modalities.

How you choose \(p(S)\) is the design lever, and it should mirror your deployment failure model, not a uniform coin flip. If lidar fails in fog once a week but the camera is essentially always up, weight \(p(S)\) toward "lidar-dropped" configurations proportionally to how much you care about performing well in them. A common, robust default is a per-modality Bernoulli drop with rate \(p_{\text{drop}}\) around \(0.15\) to \(0.3\), plus a guard that forbids the empty set (you must keep at least one modality, or the loss is undefined). When to raise the rate: when your safety case in Chapter 68 demands strong single-sensor fallback. When to lower it: when modalities are genuinely reliable and you would rather spend capacity on peak accuracy.

import torch, torch.nn as nn

class ModalityDropout(nn.Module):
    """Randomly drop whole modalities during training; never drop them all."""
    def __init__(self, drop_rate=0.25):
        super().__init__()
        self.drop_rate = drop_rate

    def forward(self, feats, learned_absent):
        # feats: (M, B, D) stacked modality features; learned_absent: (M, D)
        if not self.training:
            return feats, torch.ones(feats.shape[:2], device=feats.device)
        M, B, D = feats.shape
        keep = (torch.rand(M, B, device=feats.device) > self.drop_rate)
        # guarantee at least one surviving modality per sample
        empty = keep.sum(dim=0) == 0
        keep[torch.randint(M, (empty.sum(),)), empty.nonzero(as_tuple=True)[0]] = True
        mask = keep.float().unsqueeze(-1)                      # (M, B, 1)
        # replace a dropped modality with its LEARNED "absent" token, not zero
        feats = feats * mask + learned_absent.unsqueeze(1) * (1 - mask)
        return feats, keep.float()

Listing 50.4 shows the mechanism, and the returned keep mask is the bridge to the next idea: how the architecture should consume an absent modality once dropout has hidden it.

Mechanics of a missing input: mask, do not impute a zero

There are three honest ways to represent an absent modality, and the wrong one silently poisons the fusion. Zero-imputation, already condemned above, injects a fake measurement. Learned absent embeddings replace the missing features with a trainable vector, the same trick BERT uses for its mask token; the network learns what "I have no lidar right now" should look like in feature space, and because dropout showed it that token constantly, it responds by leaning on the survivors. Attention masking is cleaner still for transformer fusion (Section 50.2): you never fabricate a feature at all, you simply exclude the missing modality's tokens from the attention key set, so downstream queries attend only over present evidence. The softmax renormalizes over what remains, which is precisely the graceful behavior you want.

Prefer attention masking when your fusion is token-based, because it is exact: an absent modality contributes literally nothing rather than a learned approximation of nothing. Prefer learned absent embeddings when your fusion concatenates fixed-width feature vectors and a mask would break the tensor shape. Reserve zero-imputation for the case where a channel within a modality is missing and zero is a physically meaningful "no signal," which is rare. In all cases, pass the presence mask forward so the confidence head can widen its uncertainty when evidence is thin, closing the loop with the calibration machinery of Chapter 18.

import torch.nn as nn
attn = nn.MultiheadAttention(embed_dim=256, num_heads=8, batch_first=True)
# key_padding_mask: True where a modality token is ABSENT and must be ignored
fused, _ = attn(query, kv_tokens, kv_tokens, key_padding_mask=~keep_bool)

Graceful degradation as a measurable contract

Graceful degradation means the accuracy-versus-available-sensors relationship is a gentle slope, ideally monotone: removing a modality never helps, and each loss costs a bounded, predictable amount. This is a contract you can and should measure, not a hope. Build a degradation table: for every subset of modalities you evaluate the leakage-safe test performance (the held-out discipline of Chapter 65 applies here without exception), and you check two things. First, monotonicity: does any full subset underperform one of its own sub-subsets? If so, a modality is actively harmful when present, a red flag that fusion has learned a spurious dependence. Second, floor: is the worst single-sensor row still above your safety threshold? That row is your true fallback capability.

A second technique sharpens the floor: cross-modal knowledge distillation. Train a full-modality teacher, then distill it into students that see only reduced subsets, matching the teacher's soft predictions. The student inherits some of the teacher's cross-modal reasoning even though it can no longer observe the missing sensor, because the teacher's outputs encode structure the student learns to reproduce from correlated survivors. This is how a camera-only fallback can perform better than a camera-only model trained from scratch: it was taught by a model that also saw lidar.

What's Next

In Section 50.5, we pursue the frontier hint above: contrastive multimodal pretraining that maps every sensor into one shared embedding space. Once modalities live in a common geometry, a missing one is no longer a void to be masked but a gap the aligned survivors can help fill, and the same alignment powers zero-shot transfer across sensor types.