Deployment tradeoffs

"A fusion model that needs three synchronized sensors and a datacenter is not a perception system. It is a demo with good lighting."

A Deployment-Scarred AI Agent

This section assumes you are comfortable with the fusion building blocks from Sections 50.1 through 50.4 and with the missing-modality machinery in particular. It leans on edge optimization ideas (quantization, pruning, distillation) developed in Chapter 59, on the streaming and latency-budget framing of Chapter 60, and on the fleet-operations perspective of Chapter 69. If synchronization jitter is unfamiliar, revisit Chapter 3 before reading the calibration discussion below.

The four axes: accuracy, latency, energy, and unit cost

A deployment decision is a point in a four-dimensional tradeoff space. Accuracy is what the benchmark measures. Latency is the wall-clock time from sensor sample to decision, which matters because a perception result that arrives after the actuator has committed is worthless. Energy per inference sets battery life on a wearable and thermal headroom on a robot. Unit cost is the amortized dollar price of the sensors and compute multiplied across every device you will ever ship. These axes conflict: the cheapest way to raise accuracy is usually to add a modality or a fusion layer, and both cost latency, energy, and dollars.

The reason fusion is expensive is structural. If modality \(m\) produces a token sequence of length \(L_m\), a full cross-attention fusion over the concatenation costs on the order of

where \(d\) is the model width. Adding a fourth sensor does not add a quarter to the cost; because of the quadratic term it can nearly double it. This single fact explains most of the architectural moves in real systems: late fusion (combine per-modality decisions, so cost grows linearly), token pruning before fusion, and restricting attention to a small learned set of fusion queries so the quadratic term is \(k \cdot \sum_m L_m\) with \(k\) fixed rather than \((\sum_m L_m)^2\). The design question is never "which fuses best" in isolation; it is "which fuses best per millisecond and per milliwatt on my target chip."

Sensor cost is the dominant term, not FLOPs

Engineers instinctively optimize the neural network because that is the part they wrote. In shipped sensor products the network is rarely the expensive part. A lidar can cost more than the entire compute board; a medical-grade sensor carries a regulatory burden that dwarfs any GPU bill; a second camera adds a lens, an ISP channel, a calibration step on the assembly line, and a cleaning obligation for the life of the product. The right mental model is a cost-normalized accuracy: for each candidate sensor set \(S\), estimate

where \(\text{Cost}(S)\) folds in unit price, energy, bandwidth, and the operational tax of keeping that sensor calibrated and clean, and \(\lambda\) encodes how much a point of accuracy is worth in your domain. The missing-modality robustness you built in Section 50.4 is what makes this optimization actionable: if the model degrades gracefully when a sensor is absent, you can ship a cheaper sensor set to the low tier and reserve the full set for the premium tier, training one model for both. Without graceful degradation, every sensor is load-bearing and every tier needs its own model.

Dynamic fusion: pay for modalities only when they help

Static architectures pay the full fusion cost on every frame even when one cheap modality already settles the decision. Dynamic (input-adaptive) fusion breaks that. A lightweight gating policy, related to the mixture-of-experts routing in Section 50.3, decides per input whether an expensive branch is worth running. On a clear, well-lit frame the camera branch alone may be confident, so the lidar encoder and the fusion layers are skipped and both latency and energy drop. On a foggy or ambiguous frame the policy wakes the expensive branch. The expected cost becomes \(\mathbb{E}[C] = \sum_m p_m \, C_m\), where \(p_m\) is how often branch \(m\) actually fires, and in benign conditions the \(p_m\) for costly branches are small.

The tradeoff is a governance one: a data-dependent compute path has data-dependent latency, which functional-safety reviewers dislike because they must certify the worst case, not the average. The engineering answer is a hard latency budget with a guaranteed cheap fallback path, so the branch either returns in time or is preempted and the system uses the fast-modality result. That safety-case framing is developed in Chapter 68.

import torch, torch.nn as nn

class BudgetedFusion(nn.Module):
    """Skip the expensive branch when the cheap branch is already confident."""
    def __init__(self, cheap_enc, costly_enc, fuse, head, tau=0.85):
        super().__init__()
        self.cheap_enc, self.costly_enc = cheap_enc, costly_enc
        self.fuse, self.head, self.tau = fuse, head, tau

    def forward(self, x_cheap, x_costly):
        z_cheap = self.cheap_enc(x_cheap)              # always runs
        logits  = self.head(z_cheap)
        conf    = logits.softmax(-1).max(-1).values     # per-sample confidence
        run_costly = conf < self.tau                    # only the unsure samples
        used = 0
        if run_costly.any():                            # gate the costly path
            z2 = self.costly_enc(x_costly[run_costly])
            fused = self.fuse(z_cheap[run_costly], z2)
            logits[run_costly] = self.head(fused)
            used = int(run_costly.sum())
        return logits, used                             # 'used' = costly calls made

The snippet above is deliberately minimal so the mechanism is visible: confidence below tau is the only thing that spends the lidar-scale compute. In production you would replace the confidence heuristic with a learned, calibrated router and enforce a hard timeout, but the accounting stays the same.

The operational tax: synchronization, calibration, and fleet drift

The costs above are per-frame. The costs that actually sink deployments are per-fleet and per-year. A fusion model trained on well-synchronized data implicitly assumes the sensors stay aligned in time and space. In the field they do not. Clocks drift, so a 20 ms timestamp skew between camera and IMU smears a moving object across frames and quietly poisons cross-attention; this is exactly the timing problem from Chapter 3. Extrinsic calibration decays as a robot vibrates or a car flexes in the heat, so the projection that lined lidar points onto image pixels at the factory is wrong six months later. Different production batches of the "same" sensor have different noise floors and spectral responses. Early fusion is the most sensitive to all of this because it binds the modalities before any per-modality sanity check can catch the drift; late fusion is the most forgiving.

Two practices keep the operational tax bounded. First, monitor per-modality health at inference and route around a modality whose statistics have drifted out of distribution, using the same graceful-degradation path you trained for missing data; the distribution-shift detection in Chapter 66 supplies the trigger. Second, evaluate deployment tradeoffs on a leakage-safe split (Chapter 5): if the same drive, subject, or device leaks across train and test, your latency-versus-accuracy curve is fiction and you will overspend on a sensor that only looked useful because the test set was contaminated.

What's Next

In Chapter 51, we move from fusing sensors into a decision to fusing them into a reconstructed world. Neural fields and Gaussian splatting build an explicit 3D representation from multi-sensor streams, which lets you re-simulate a lidar or camera view, generate synthetic data for the rare events your deployment tradeoffs left underdefined, and reason about a scene rather than a single frame.