Modality-specific encoders

"You would not read a barcode and a sonnet with the same eye. Do not ask one network to."

A Specialized AI Agent

One encoder per modality, and why not one for all

The design rule is blunt: give every modality its own encoder, tuned to that modality's structure, and make them all agree on only one thing, the width \(d\) of the vectors they emit. The what is a set of functions \(f_m: \mathcal{X}_m \to \mathbb{R}^{n_m \times d}\), one per modality \(m\), each turning its native input into \(n_m\) tokens of dimension \(d\). The why is inductive bias. A convolution assumes translation-equivariant local structure, which is true of images and spectrograms and false of a bag of lidar points. A permutation-invariant set encoder is right for that point cloud and wrong for a time series, where order is the signal. Forcing one architecture across modalities means every modality but one is being processed under an assumption it violates.

The how is to reuse the right backbone from earlier parts of the book. Images and depth maps go through a CNN or vision transformer; point clouds and lidar sweeps through a set or voxel network of the kind built in Chapter 42; inertial, audio, and biosignal streams through the 1D convolutional or transformer encoders of Chapter 13; a handful of scalar tags through a small MLP. The when to break the rule is worth stating precisely: share an encoder only when the streams are genuinely the same modality, three identical accelerometers on one body, a stereo pair of the same camera, where a shared \(f_m\) both saves parameters and enforces a consistent representation. Distinct physics means distinct encoders.

Making heterogeneous streams commensurable

Three concrete mismatches have to be resolved inside the encoders, or they leak into fusion as bugs. The first is dimensionality: each encoder ends in a projection to the shared width \(d\), typically a linear layer on a pooled or per-token feature, so a 2048-channel CNN feature and a 128-channel IMU feature both leave as \(d\)-vectors. The second is token count. An encoder can emit one summary vector per modality (\(n_m = 1\), cheap, discards structure) or a sequence of tokens (\(n_m > 1\), preserves where and when, which the cross-attention of Section 50.2 needs). The choice is an accuracy-versus-cost knob you set here.

The third and most treacherous is rate and phase. Modalities arrive at wildly different sampling rates and with real time offsets, exactly the synchronization problem of Chapter 3. Two disciplines keep this honest. Encode each stream over a shared time window rather than a shared sample count, so a 200 Hz IMU and a 30 Hz camera covering the same second stay aligned in meaning even though their token counts differ. And normalize per modality: an accelerometer in \(\text{m/s}^2\), a pixel in \([0,1]\), and a temperature in kelvin cannot share one normalization statistic, so each encoder owns its own input scaling. On the leakage-safe discipline of Part I, these normalization statistics are fit on training data only; a normalizer fit across the whole set silently leaks test-time distribution into every modality at once.

import torch, torch.nn as nn

D = 256   # shared embedding width every modality must emit

class IMUEncoder(nn.Module):        # 6-axis inertial, (B, 6, T) at high rate
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv1d(6, 64, 7, stride=2, padding=3), nn.GELU(),
            nn.Conv1d(64, 128, 5, stride=2, padding=2), nn.GELU(),
            nn.AdaptiveAvgPool1d(1))          # one summary token
        self.proj = nn.Linear(128, D)
    def forward(self, x):
        return self.proj(self.net(x).squeeze(-1)).unsqueeze(1)   # (B, 1, D)

class ImageEncoder(nn.Module):      # camera frame, (B, 3, H, W)
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv2d(3, 32, 3, stride=2, padding=1), nn.GELU(),
            nn.Conv2d(32, 64, 3, stride=2, padding=1), nn.GELU(),
            nn.AdaptiveAvgPool2d(2))          # a 2x2 grid = 4 tokens
        self.proj = nn.Linear(64, D)
    def forward(self, x):
        f = self.net(x).flatten(2).transpose(1, 2)   # (B, 4, 64)
        return self.proj(f)                           # (B, 4, D)

imu  = IMUEncoder()(torch.randn(8, 6, 200))     # (8, 1, 256)
cam  = ImageEncoder()(torch.randn(8, 3, 64, 64))# (8, 4, 256)
tokens = torch.cat([imu, cam], dim=1)           # (8, 5, 256) ready for fusion
print(imu.shape, cam.shape, tokens.shape)

As Listing 50.1 shows, the interface between "sensing" and "fusion" is exactly this shape agreement. Everything specific to a sensor, its geometry, its rate, its physics, is spent inside its own encoder; everything after the concatenation is modality-blind. That clean seam is what lets you swap a better image backbone in without touching the fusion head.

Pretrained backbones and where to freeze

Encoders rarely start from scratch. The single biggest lever on multimodal accuracy is initializing each \(f_m\) from a strong single-modality model, an ImageNet or CLIP vision backbone, an audio encoder, a self-supervised inertial model from Chapter 17, then training only a thin projection plus the fusion layers on top. The why is data economy: paired multimodal data is scarce and expensive, while unimodal pretraining data is abundant, so you want most of each encoder's competence to come from cheap single-modality learning. The when to freeze versus fine-tune is a small decision tree: freeze the backbone and train only the projection when paired data is tiny or the modality is well covered by its pretrained model; unfreeze the top blocks when your sensor's domain (a wrist accelerometer, a thermal camera) drifts far from the backbone's training distribution. Either way you keep the projection head trainable, because that is the layer that rotates each modality into the shared space fusion expects.

import timm, torch.nn as nn
backbone = timm.create_model("vit_small_patch16_224", pretrained=True, num_classes=0)
img_encoder = nn.Sequential(backbone, nn.Linear(backbone.num_features, 256))  # -> (B, 256)

What's Next

In Section 50.2, we take the aligned token sequences these encoders produce and let the modalities actually talk to each other, using cross-attention and transformer fusion to route information between streams rather than merely concatenating them, so that the camera tokens can be sharpened by the radar tokens and back again.