Early, middle, and late fusion

"You keep asking where I combine the sensors. I keep telling you: that is the only question you have actually asked me."

A Well-Architected AI Agent

The three depths, defined by where the streams meet

Picture a pipeline as a sequence: raw signal, features, representation, decision. Fusion is the point where two or more of these pipelines join into one. The taxonomy is nothing more than naming that junction by how early it happens.

Early fusion (also called data-level or input-level fusion) concatenates the raw or lightly-preprocessed streams into one combined input and hands that to a single model. If sensor \(A\) yields a vector \(x_A\) and sensor \(B\) yields \(x_B\), early fusion models \(p(y \mid [x_A ; x_B])\) with one joint function. The why is expressive power: a single model sees both streams simultaneously and can learn arbitrary cross-sensor interactions, including subtle ones no human would hand-code. The when is when the sensors are tightly synchronized and spatially registered, so that concatenating sample \(t\) of \(A\) with sample \(t\) of \(B\) is physically meaningful. Pixel-aligned RGB and depth, or the three axes of a single accelerometer, are natural early-fusion inputs.

Late fusion (decision-level fusion) sits at the opposite extreme. Each sensor runs its own complete model to a prediction, and only those predictions are combined: \(y = g\big(f_A(x_A),\, f_B(x_B)\big)\), where \(g\) might be an average, a weighted vote, or a small learned combiner. The why is modularity and robustness. Each branch trains independently, can be swapped or upgraded on its own, and if one sensor dies you simply drop its vote and keep running. The cost is that the branches never talk during perception, so any information that lives in the joint pattern (the way two streams move together) is lost before the merge.

Middle fusion (feature-level or intermediate fusion) is the pragmatic middle ground that dominates modern multimodal deep learning. Each sensor is encoded into a learned intermediate representation, those representations are combined (concatenated, added, attended over, or gated), and a shared head reasons over the fused embedding: \(y = h\big(\phi(e_A, e_B)\big)\) with \(e_A = \mathrm{enc}_A(x_A)\). The why is that it keeps most of early fusion's cross-modal expressiveness while letting each encoder specialize to its sensor's statistics, and it tolerates modest misalignment because features are more forgiving than raw samples. Figure 48.2.1 places all three on a single axis.

What each depth costs you, concretely

The three depths are not merely stylistic; each imposes hard, predictable consequences on four things you care about: expressiveness, robustness to a failed sensor, training and data cost, and synchronization tolerance.

Expressiveness. Early fusion is the most expressive because nothing is discarded before the joint model sees it, but that same freedom is a curse: with high-dimensional raw inputs (think a lidar point cloud plus a camera frame), the concatenated space is enormous, and the model needs far more data to avoid overfitting spurious cross-sensor coincidences. Late fusion is the least expressive: by the time you see \(f_A(x_A)\), the raw joint structure is gone, so a combiner that averages two probabilities can never recover a pattern that only existed in the raw pair. Middle fusion lets you tune this: a fat fusion layer keeps more joint structure, a thin one keeps less.

Robustness to missing modalities. This is where late fusion shines and early fusion suffers. If sensor \(B\) drops out, a late-fusion system just stops counting \(B\)'s vote. An early-fusion model, by contrast, was trained on a concatenated input of a fixed shape; hand it zeros for \(x_B\) and it may behave unpredictably, because it never learned that a flat channel means "absent" rather than "reading zero." Handling this properly is a whole topic of its own, taken up in Section 48.4 and pushed to deep architectures in Chapter 50.

Synchronization tolerance. Early fusion demands the tightest temporal and spatial alignment, because you are pairing raw samples; a few milliseconds of skew between a microphone and a camera can wreck a lip-reading model. Late fusion is the most forgiving, since each branch consumes its own stream at its own rate and only the final decisions meet. This alignment burden, and the calibration that earns it, is the subject of Section 48.3 and rests on the timing foundations from Chapter 3.

Table 48.2.1 summarizes the tradeoffs so you can pick a starting point by inspection rather than by trial and error.

A minimal implementation of all three

Nothing about the taxonomy is mysterious in code. Listing 48.2.1 implements early, middle, and late fusion for two toy sensor encoders in PyTorch, so you can see that the entire distinction is where the tensors are concatenated relative to the encoders and the head. Read the three forward methods side by side: early concatenates before any encoding, middle concatenates the encoder outputs, late concatenates the per-branch logits.

import torch, torch.nn as nn

class Encoder(nn.Module):            # a stand-in per-sensor feature extractor
    def __init__(self, d_in, d_out):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(d_in, 32), nn.ReLU(), nn.Linear(32, d_out))
    def forward(self, x): return self.net(x)

class EarlyFusion(nn.Module):
    def __init__(self, dA, dB, n_cls):
        super().__init__()
        self.model = Encoder(dA + dB, n_cls)          # one model over concatenated raw input
    def forward(self, xA, xB): return self.model(torch.cat([xA, xB], dim=-1))

class MiddleFusion(nn.Module):
    def __init__(self, dA, dB, n_cls, d=16):
        super().__init__()
        self.encA, self.encB = Encoder(dA, d), Encoder(dB, d)
        self.head = nn.Linear(2 * d, n_cls)           # reason over fused embeddings
    def forward(self, xA, xB):
        return self.head(torch.cat([self.encA(xA), self.encB(xB)], dim=-1))

class LateFusion(nn.Module):
    def __init__(self, dA, dB, n_cls):
        super().__init__()
        self.mA, self.mB = Encoder(dA, n_cls), Encoder(dB, n_cls)  # full model per sensor
    def forward(self, xA, xB, wA=0.5):
        return wA * self.mA(xA).softmax(-1) + (1 - wA) * self.mB(xB).softmax(-1)

As Listing 48.2.1 makes visible, "which fusion" is a one-line architectural choice, not a different framework. That is exactly why it deserves a deliberate decision rather than a default: the code cost is identical, but the behavioral consequences in Table 48.2.1 are not.

from torchmultimodal.modules.fusions.attention_fusion import AttentionFusionModule
# fuses a dict of per-sensor embeddings via learned attention weights
fusion = AttentionFusionModule(channel_to_encoder_dim={"imu": 64, "ppg": 64},
                               encoding_projection_dim=64)
fused = fusion({"imu": e_imu, "ppg": e_ppg})   # one call handles the weighted merge

Hybrids, and why the boundaries blur

Real systems rarely sit at a single pure depth. A self-driving stack might fuse camera and lidar in the middle to build a shared bird's-eye-view representation (the theme of Chapter 43), then fuse that perception module's output with a separate radar tracker (Chapter 44) at the decision level, because radar's independent failure modes make it a natural late-fusion safety vote. That is early-or-middle and late in one pipeline, chosen per sensor pair by asking, for each junction, the key-insight question: does the label live in the joint pattern here, or in each stream alone?

The deeper reason the boundaries blur is that "raw," "feature," and "decision" are points on a continuum, not three discrete bins. A learned encoder's early layers already produce feature-like activations, so an early-fusion model with per-channel input stems is creeping toward middle fusion. Treat the taxonomy as a compass, not a cage: it tells you which way expressiveness, robustness, and alignment cost move as you slide the merge point, and that directional intuition is what you actually deploy.

What's Next

Every depth in this section quietly assumed that sample \(t\) of one sensor lines up with sample \(t\) of another. That assumption is a lie the hardware tells you, and it is expensive to make true. In Section 48.3, we tackle temporal alignment and calibration between sensors: how clocks drift apart, how to time-register streams sampled at different rates, and how spatial calibration earns the pixel-level pairing that early fusion silently requires.