"You asked me to learn from a crowd where some carry a watch, some carry earbuds, and a few carry both. I stopped demanding everyone bring the same senses and let each teach me what it could."
An Accommodating AI Agent
Prerequisites
This section assumes you have the federated averaging loop from Section 64.2 firmly in hand, and that you have met deep multimodal fusion and the problem of missing modalities in Chapter 50, built on the fusion foundations of Chapter 48. Cross-modal alignment leans on the contrastive learning ideas of Chapter 17. The mathematics is again elementary: weighted averages, this time computed per subnetwork rather than over one monolithic weight vector.
The Big Picture
Every federated method so far assumed a comforting uniformity: every client trains the same model on the same kind of data. Reality is messier. A wearable fleet is a menagerie of sensor suites. One user carries a phone with an IMU, another adds a watch with PPG and an ECG lead, a third pairs earbuds with a microphone and a barometer. The devices differ, the modalities differ, and often several devices belong to one person and should cooperate. This section handles two intertwined kinds of heterogeneity that plain FedAvg cannot: modality heterogeneity, where clients hold different sensor channels, and cross-device federation, where a single learner is spread across a personal constellation of gadgets. The unifying trick is to stop treating the model as one indivisible blob to average. Split it into modality-specific encoders and a shared fusion head, then federate each piece only among the clients that own it. When you finish, you will be able to design a federated system that learns one shared representation from a fleet no two of whose members carry the same senses.
Two axes of heterogeneity: which features, whose device
Standard federated learning is horizontal: clients hold different samples but the same feature space, like a thousand watches each recording the same accelerometer axes for different wearers. Multimodal federation breaks that assumption along a second axis. In the vertical setting, different parties hold different features for overlapping subjects, the classic example being a hospital with ECG and a fitness app with step counts for the same enrolled patient. In sensing the dominant and more tractable case is a hybrid: a fleet of devices, each holding its own subjects and its own subset of modalities. Client \(k\) owns a modality set \(M_k \subseteq \mathcal{M}\), and \(M_k\) varies across the fleet. The why it matters: if you naively demand a common input format, you must discard every channel that any client lacks, collapsing to the intersection \(\bigcap_k M_k\), which for a diverse consumer fleet is often just the IMU. You throw away the PPG, audio, and barometer signal precisely because it is unevenly distributed. The goal of multimodal federation is to keep the union \(\bigcup_k M_k\) in play while never requiring any one client to hold all of it.
Cross-device federation adds a physical wrinkle. "One client" need not be one gadget. A person's phone, watch, and earbuds form a personal-area network that can sense the same event from complementary vantage points at the same instant. Treating them as a single logical client lets their modalities fuse, but it demands the tight time synchronization of Chapter 3: fusing a heartbeat from the watch with a cough from the earbuds is meaningless if their clocks disagree by half a second.
Key Insight
The move that makes multimodal federation work is modular parameter sharing. Give each modality \(m\) its own encoder \(\phi_m\) and share one fusion-and-head network \(\psi\) across the whole fleet. Now aggregation is done per module, not per model: encoder \(\phi_m\) is averaged only over the clients that actually carry modality \(m\), while the fusion head \(\psi\) is averaged over everyone. A client missing a modality simply does not vote on that encoder, and it never has to fake data it did not collect. The single global weight vector of FedAvg becomes a dictionary of independently federated parts, and modality heterogeneity turns from a blocker into a bookkeeping problem.
Federating a model made of parts
Concretely, client \(k\) computes a fused representation only from the modalities it owns,
\[ z_k = \psi\!\Big(\big\{\, \phi_m(x_m) : m \in M_k \,\big\}\Big), \]where \(\psi\) pools a variable-size set of per-modality embeddings (mean-pool, attention pool, or a gated sum) so it accepts any subset. After a round of local training, the server aggregates each encoder over its owning cohort \(S_m = \{k : m \in M_k\}\):
\[ \phi_m \leftarrow \sum_{k \in S_m} \frac{n_k}{\sum_{j \in S_m} n_j}\, \phi_m^{(k)}, \qquad \psi \leftarrow \sum_{k} \frac{n_k}{\sum_j n_j}\, \psi^{(k)}. \]The how of robustness is modality dropout during local training: even clients that own several modalities should randomly hide some each step, so the shared fusion head \(\psi\) learns to produce a usable \(z\) from any subset rather than leaning on a channel that half the fleet lacks (the same missing-modality discipline as Chapter 50). A second concern is representation drift between modalities: if the IMU cohort and the PPG cohort never co-train, their embeddings can land in incompatible regions of latent space and the fusion head cannot combine them. The fix is a cross-modal alignment loss on the clients that do hold pairs, pulling co-occurring modality embeddings together (a federated cousin of the contrastive objective in Chapter 17). Those multi-modality clients act as anchors that stitch the single-modality cohorts into one shared space.
The listing below implements modality-aware aggregation for a fleet of three client profiles. It is the structural heart of this section and is discussed immediately after.
import numpy as np
from collections import defaultdict
# Each client owns a subset of modalities and a local copy of the parts it holds.
FLEET = [
{"mods": ["imu"], "n": 300}, # phone only
{"mods": ["imu", "ppg"], "n": 500}, # phone + watch (an anchor)
{"mods": ["ppg", "audio"], "n": 200}, # watch + earbuds
]
def local_update(part, seed): # stand-in for one round of local SGD
return part + 0.05 * np.random.default_rng(seed).standard_normal(part.shape)
# Global modules: one encoder per modality, one shared fusion head.
enc = {m: np.zeros(4) for m in ["imu", "ppg", "audio"]}
fusion = np.zeros(4)
for rnd in range(3):
enc_bucket = defaultdict(list) # (weight, vector) per modality
fus_bucket = []
for cid, c in enumerate(FLEET):
for m in c["mods"]: # train only owned encoders
enc_bucket[m].append((c["n"], local_update(enc[m], seed=rnd * 10 + cid)))
fus_bucket.append((c["n"], local_update(fusion, seed=99 - cid)))
# Each encoder is averaged only over its owning cohort; fusion over everyone.
for m, ups in enc_bucket.items():
tot = sum(w for w, _ in ups)
enc[m] = sum(w * v for w, v in ups) / tot
tot = sum(w for w, _ in fus_bucket)
fusion = sum(w * v for w, v in fus_bucket) / tot
print("audio encoder saw", len([c for c in FLEET if "audio" in c["mods"]]), "of",
len(FLEET), "clients; fusion head saw all", len(FLEET))As Listing 64.6.1 shows, the entire mechanism is a change of aggregation granularity. FedAvg averaged one vector; here we average a bucket per module and let the ownership sets \(S_m\) decide who votes on what. The audio encoder, held by a single client, learns more slowly and generalizes less; that imbalance in per-modality "coverage" is the multimodal analogue of the label skew from Section 64.2, and it is why rare modalities benefit most from the alignment anchors.
Real-World Application: cross-device federation on a hearables and watch platform
A consumer health platform ships earbuds (microphone, in-ear PPG, IMU) and a watch (wrist PPG, ECG, IMU) that customers mix and match: some own both, most own one. The team wanted a single shared cough-and-respiration model without pooling any audio or cardiac data on a server. They federated a per-modality encoder for each channel and one attention-fusion head, trained with modality dropout so the head worked whether or not audio was present. Clients who owned both devices ran a contrastive alignment loss between the earbud microphone and the watch PPG, anchoring the audio and cardiac spaces so watch-only users still benefited from acoustic knowledge they never contributed. The devices in a paired setup synchronized to within a few milliseconds over Bluetooth before fusing, honoring the timing requirement of Chapter 3. Watch-only users, who had been stuck with a PPG-only baseline, gained measurable respiration-rate accuracy from a model that had never seen their audio and never could.
What survives from single-modality federation, and what breaks
Everything you learned about non-IID data still applies, per module. Each encoder cohort \(S_m\) has its own feature skew and its own straggler behavior, so FedProx or SCAFFOLD-style drift correction (Section 64.2) is applied independently to each part. Personalization (Section 64.3) composes cleanly: keep the modality encoders global and personalize only the fusion head per user, or vice versa. What genuinely breaks is evaluation and privacy accounting. A federated multimodal model must be scored on clients across the full spectrum of modality availability, not just the well-equipped ones, or you will ship a system that quietly fails for the majority who own a single device. And the privacy budget is now per module: the audio encoder, seen by few clients, leaks more per contribution than the IMU encoder seen by thousands, so the differential-privacy and secure-aggregation machinery of Section 64.4 must account for uneven cohort sizes. Multimodal foundation encoders for wearables (Chapter 20) make attractive frozen backbones here, letting you federate only a small fusion head and cut per-round traffic to a trickle.
The Right Tool
You do not implement the per-module bookkeeping, client sampling, and transport by hand. Flower's strategies aggregate whatever parameter arrays a client returns, so you can wrap each module as its own federated group: a client returns only the arrays for the modules it owns, and a custom configure_fit plus a keyed aggregation function does the rest.
import flwr as fl
class ModalClient(fl.client.NumPyClient):
def __init__(self, owned): self.owned = owned # e.g. ["imu", "ppg"]
def fit(self, params, config):
load_owned_modules(model, params, self.owned)
train_one_round(model, local_loader, modality_dropout=0.3)
# return ONLY the modules this client holds, tagged by name
return dump_owned_modules(model, self.owned + ["fusion"]), n_local, \
{"mods": ",".join(self.owned)}
# A custom strategy averages each named module over just its owning clients.
fl.simulation.start_simulation(
client_fn=lambda cid: ModalClient(owned=FLEET_MODS[cid]),
num_clients=1000,
strategy=ModalityAwareFedAvg(), # keys updates by module name before averaging
)Listing 64.6.2 removes the plumbing, not the design. Which modules to share versus personalize, how aggressively to drop modalities, and where to place the alignment loss remain experiments you own.
Research Frontier
The active frontier is federating across incompatible modality sets without any paired data at all. FedMSplit and CreamFL train a shared fusion or knowledge-transfer layer while each client keeps modality-private encoders, exchanging class prototypes or public-data logits rather than weights so that an audio-only client and an IMU-only client can still align through a common label space. Vertical federated learning with split networks (see Section 64.5) is being pushed toward the case where different institutions hold different sensor modalities for the same patients, with secure aggregation over the cut layer. A third thread federates multimodal sensor-language models (Chapter 21), exchanging only lightweight adapters so a fleet of heterogeneous wearables can jointly fine-tune a reasoning model over their private, never-pooled sensor streams.
Exercise
Extend Listing 64.6.1 so that clients holding two modalities also emit a scalar "alignment gap," the mean squared distance between their two modality embeddings on a shared batch, and have the server average it. Now vary the fraction of two-modality anchor clients from 0 to 0.5 of the fleet and track the gap over rounds. Confirm the prediction of this section: with no anchors the single-modality encoders drift into incompatible spaces (large, non-decreasing gap), and even a small anchor fraction pulls them together. Report the minimum anchor fraction that keeps the gap shrinking, and relate it to the audio encoder's low coverage.
Self-Check
1. Distinguish horizontal, vertical, and the hybrid device-plus-modality heterogeneity setting, giving one sensing example of each.
2. Why is a rare modality's encoder more exposed under differential privacy than a common one, and what does that imply for the per-module privacy budget?
3. What role do two-modality "anchor" clients play, and what fails in the shared representation if the fleet contains none of them?
What's Next
In Section 64.7, we confront the uncomfortable truth that a server which never sees the data also cannot easily see when the system is broken, attacked, or lying to itself. We turn to federated evaluation, security, and failure modes: how to measure a model whose test set is scattered across devices, defend the aggregation against poisoned updates, and recognize the ways a federated sensor system fails silently.