Fusion under missing modalities

"I was trained on five senses and shipped with three. The other two are a rumor I tell myself at inference time."

A Resigned AI Agent

This section assumes you understand the fusion stages from Section 48.2 (early, middle, late) and the complementary-vs-redundant distinction from Section 48.1. The probability notation for marginalization draws on the primer in Chapter 4. The deep, learned approaches to this problem (cross-attention with modality tokens, gated mixtures of experts, contrastive pretraining that survives dropout) are the subject of Chapter 50; here we stay at the level every fused system needs regardless of architecture.

Why modalities go missing, and why the reason matters

Before choosing a coping strategy, classify the failure. Borrowing the missing-data taxonomy from statistics and adapting it to sensing gives three regimes with sharply different consequences.

• Missing completely at random (MCAR). A packet drops, a bus arbitration collision loses a frame, a sensor reboots. The absence carries no information about the scene. Simple imputation is safe here.
• Missing at random (MAR). The absence correlates with other observed signals but not with the hidden truth given those signals. Example: a wearable's photoplethysmography (PPG) drops out during high accelerometer energy (vigorous motion), and you still have the accelerometer. You can predict and compensate.
• Missing not at random (MNAR). The absence correlates with the very thing you want to infer. A radar that stops returning because the target is now occluded, or a glucose sensor that fails because of the extreme physiology you care about. This is the dangerous case: a naive imputer will hallucinate normality precisely when the world is abnormal.

Four core strategies, from crude to principled

Given a set of modalities \(\{x_1, \dots, x_M\}\) and an availability mask \(m \in \{0,1\}^M\), there are four load-bearing ways to produce an output when some \(m_i = 0\).

1. Drop and re-weight (late fusion). When each modality has its own head producing a prediction or a score, simply omit the absent branches and renormalize the combination. If you fuse per-modality posteriors, dropping a term and renormalizing over the survivors is exactly marginalization under an independence assumption. This is why late fusion degrades most gracefully under missingness: the architecture never assumed a fixed-width concatenated vector. The cost is that late fusion cannot exploit fine cross-modal interactions (see the tradeoff in Section 48.2).

2. Masked / default imputation. Replace the missing input with a learned or statistical default (the training-set mean, a zero after mean-centering, or a per-context prior) and pass the mask alongside so the model knows the value is synthetic. Cheap and effective for MCAR. Actively harmful for MNAR, because the default asserts "typical" exactly when reality is atypical.

3. Cross-modal reconstruction. Learn to predict the missing modality from the present ones, then fuse the reconstruction. This works when modalities are redundant: an inertial stream can stand in for a dropped visual-odometry frame over short gaps (the dead-reckoning idea from Chapter 24). It fails for complementary modalities that carry non-overlapping information, since there is nothing to reconstruct from.

4. Marginalization (probabilistic fusion). In a Bayesian estimator you integrate the absent measurement out of the posterior rather than inventing a value:

A Kalman-family filter does this almost for free: when a sensor is unavailable at time \(t\), you run the prediction step and skip that measurement update, and the covariance simply grows to reflect the increased uncertainty. This is the cleanest answer and the reason state-estimation fusion (Chapter 49) handles dropouts so naturally. The uncertainty growth is honest: the system knows it knows less.

Training so the model expects gaps

A model only degrades gracefully at inference if it saw degradation during training. The workhorse technique is modality dropout: during training, randomly zero out entire modalities (with their masks set) so the network learns to produce sensible outputs from any subset. This is the multimodal analogue of ordinary dropout and it is what makes late-fusion and attention-fusion models robust rather than merely functional. Crucially, sample the dropout pattern to match your deployment missingness distribution: if the camera fails ten times more often than the IMU, weight the dropout accordingly, or the model will over-invest in a modality that is rarely the one missing.

import numpy as np

def modality_dropout(batch, drop_probs, rng):
    """Randomly null out whole modalities and return an availability mask.
    batch: dict name -> array of shape (B, ...); drop_probs: dict name -> p."""
    B = next(iter(batch.values())).shape[0]
    out, mask = {}, {}
    for name, x in batch.items():
        keep = rng.random(B) >= drop_probs[name]          # 1 = present
        # never drop every modality for a given sample:
        m = keep.astype(np.float32)
        out[name] = x * m.reshape(B, *([1] * (x.ndim - 1)))  # zero the absent ones
        mask[name] = m
    present = np.stack(list(mask.values()), axis=1).sum(1)
    for i in np.where(present == 0)[0]:                     # rescue all-missing rows
        name = rng.choice(list(batch.keys()))
        mask[name][i] = 1.0
        out[name][i] = batch[name][i]
    return out, mask                                       # feed BOTH to the model

rng = np.random.default_rng(0)
data = {"imu": np.ones((4, 6)), "ppg": np.ones((4, 1)), "cam": np.ones((4, 128))}
x, m = modality_dropout(data, {"imu": 0.1, "ppg": 0.4, "cam": 0.3}, rng)

The modality_dropout function above is deliberately framework-agnostic and about fifteen lines. In a real training loop it slots in as a batch transform, and you pair it with a loss that is only computed over present targets. Notice the two habits it enforces: the mask travels with the data, and no sample is ever left with zero modalities (which would produce a meaningless gradient).

Evaluating degradation, not just the happy path

A single accuracy number on fully-observed data hides the failure you actually ship. Report a degradation curve: performance as a function of which modalities are present, ideally one row per realistic missingness pattern rather than a single "all sensors" cell. Two systems with identical full-input accuracy can differ enormously when the camera drops, and the leakage-safe protocols of Chapter 65 apply here too: the missingness in your test set must mirror deployment, and you must not tune the dropout schedule on the test split. Section 48.6 develops the full evaluation methodology for fused systems; the point to carry forward is that missingness is an axis of your evaluation, not a footnote.

What's Next

In Section 48.5, we turn the crude "drop and renormalize" step into something principled: confidence-weighted combination, where each modality contributes in proportion to how much it should be trusted at this instant. That is exactly the machinery a fused system needs when a modality is not fully absent but merely degraded, noisy, or low-confidence.