Multispectral fusion

"Give me one band and I will guess. Give me twelve and I can tell a green leaf from green paint, and a warm rock from a person pretending to be one."

A Multispectral AI Agent

This section assumes you can read each band as a calibrated physical quantity: reflectance for the reflective bands and apparent temperature for the thermal band, the radiometric story from Section 45.1 and the sensor-model intuition of Chapter 2. Multispectral fusion is a special, physically grounded case of the general fusion machinery in Chapter 48; here the modalities are wavelengths rather than whole sensors, which changes the alignment problem and the failure modes but not the underlying early/middle/late taxonomy.

What the extra bands buy you

The value of a band is the discriminative information it carries that the others do not. Two surfaces can be metamers in RGB, identical to three broadband detectors, yet diverge sharply once you look at their reflectance at 850 nm or 1600 nm. Vegetation is the canonical example: healthy leaves absorb red and reflect NIR strongly, so the normalized difference vegetation index

separates living canopy from soil, roads, and green-painted surfaces that would fool an RGB classifier. Bands are not equally useful, and they are highly correlated: a hyperspectral cube with 200 contiguous bands typically has an effective dimensionality of a handful, so the first job is often band selection or a projection (PCA, minimum-noise-fraction) that keeps the informative axes and discards the redundant ones, exactly the dimensionality-reduction reasoning from Chapter 8. Choose bands for complementarity (they disagree usefully) and redundancy (they agree when one is degraded), the two mechanisms that make any fusion pay off.

Registration: the problem before the model

Bands rarely share a focal plane. A multispectral rig may use a filter wheel (bands captured at different times), a beam-splitter array, or entirely separate cameras for RGB and LWIR with different resolutions, fields of view, and optical centers. Before any pixel-level fusion, the bands must be co-registered so that a pixel means the same physical point across all channels. This is a warping problem: estimate a homography or a dense flow that maps each band onto a common frame. Cross-spectral registration is genuinely hard because intensity does not correspond across the visible-to-thermal gap; a bright edge in RGB may be invisible in LWIR. Practitioners key on structural cues (gradients, mutual information, edge maps) rather than raw intensity, and calibrate the extrinsics once with a target that is visible in every band (a heated checkerboard works for RGB-plus-thermal). Misregistration is the single most common reason a fused model underperforms its best single band: a two-pixel shift between NIR and red turns a clean NDVI edge into a colored fringe of false anomalies.

Early, middle, and late fusion of bands

The same three-level taxonomy from Chapter 48 applies. Early (pixel-level) fusion concatenates registered bands into an \(N\)-channel tensor and feeds a single network; it is simple and preserves fine spatial detail, but it demands near-perfect registration and forces all bands through one shared stem. Middle (feature-level) fusion runs a lightweight encoder per band, then merges the feature maps, which tolerates small misalignments and lets each band learn a native front end; this is where illumination-aware gating lives, learning to down-weight RGB at night and lean on LWIR. Late (decision-level) fusion runs a full detector per band and combines outputs, which is the most robust to a dead sensor but throws away cross-band interactions. For thermal-plus-RGB perception, middle fusion with a learned per-pixel gate is the workhorse, and it connects directly to the missing-modality robustness techniques of Chapter 50: train with band dropout so the network never assumes every wavelength is present.

The code below shows the core idea of illumination-aware weighting without a training loop. It estimates a per-pixel confidence for the reflective bands from local brightness and contrast, then blends a normalized reflective feature with a thermal feature so that dark, low-contrast regions defer to LWIR. This is the classical prior that a learned gate rediscovers, and it is worth building by hand once to see why the gate helps.

import numpy as np

def illumination_aware_fuse(rgb, nir, lwir, eps=1e-6):
    """Blend reflective and thermal features by per-pixel illumination confidence.
    rgb: HxWx3 reflectance in [0,1]; nir, lwir: HxW (lwir = apparent temp, C)."""
    lum = rgb.mean(axis=2)                                  # scene brightness proxy
    local_contrast = np.abs(lum - lum.mean())              # cheap global-contrast term
    # confidence in the visible/NIR bands: bright AND textured -> trust them
    conf_vis = np.clip(lum, 0, 1) * (0.5 + 0.5 * np.tanh(4 * local_contrast))
    conf_vis = conf_vis[..., None]

    ndvi = (nir - rgb[..., 0]) / (nir + rgb[..., 0] + eps)  # a reflective feature
    vis_feat = np.stack([rgb.mean(2), ndvi], axis=-1)       # HxWx2
    therm_feat = np.stack([(lwir - lwir.min()) /
                           (lwir.ptp() + eps)] * 2, axis=-1)  # HxWx2, normalized

    fused = conf_vis * vis_feat + (1 - conf_vis) * therm_feat
    return fused, conf_vis[..., 0]

H, W = 4, 4
rgb = np.full((H, W, 3), 0.05); rgb[:, :2] = 0.8           # left half bright, right dark
nir = np.full((H, W), 0.6);     lwir = np.full((H, W), 22.0); lwir[1:3, 2:4] = 31.0
fused, conf = illumination_aware_fuse(rgb, nir, lwir)
print(conf.round(2))            # high on the lit left, near-zero on the dark right

Evaluate fused models with the same care as any multi-sensor system. Split by capture site or session, never by random frame, so aligned band tuples from one scene cannot straddle train and test; that is the leakage-safe discipline of Chapter 5, and it bites harder here because tightly registered bands make near-duplicate leakage almost invisible. Always report the fused model against the best single-band baseline; fusion that does not beat its strongest ingredient is usually a registration or gating bug, not a data limitation.

Where fusion pays and where it does not

Multispectral fusion earns its complexity when the extra bands change the decision, not merely decorate the image. It shines in remote sensing and agriculture (material and health discrimination), automotive night perception (RGB-plus-LWIR pedestrian detection through the domain-adaptation lens of Section 45.4), medical imaging (oxygenation from spectral reflectance), and security (camouflage and spoof detection). It underperforms when bands are cheap to add but carry no independent signal, when registration cannot be held to sub-pixel accuracy, or when the added channels balloon compute past the platform budget. On an edge device, the honest tradeoff is bands versus latency: every extra channel costs bandwidth, storage, and inference time, so band selection is a deployment decision, not only a modeling one, and it connects to the on-device optimization of Chapter 59.

What's Next

In Chapter 46, we leave the intensity image behind entirely. Event cameras report per-pixel brightness changes asynchronously, at microsecond latency and enormous dynamic range, trading the dense frame for a sparse stream of events. We will see how to represent that stream, reconstruct video from it, and fuse it with frames and inertial data, a different answer to the same question this chapter has been asking: how do you perceive when the ordinary camera runs out of physics?