Personalization and local adaptation

"The fleet trained a model that is wrong about everyone equally. I kept a little of it for myself, adjusted the last layer to how this wrist actually moves, and suddenly the average stopped speaking for me."

A Self-Interested AI Agent

Why one global model is the wrong deliverable

Section 64.2 showed that under non-IID clients, the FedAvg objective minimizes a weighted average of local losses. The minimizer of an average is not the minimizer for any single term unless every term already agrees. In sensing, the terms rarely agree. Two people performing the "climbing stairs" activity produce accelerometer distributions with different means, variances, and even different dominant frequencies because of stride length, cadence, and where the phone sits in the pocket. A model forced to be good on average across both learns a decision boundary that sits between their clusters and slices through each of them.

The formal object we actually want is a family \(\{\theta_1, \dots, \theta_K\}\), one model per client \(k\), each minimizing that client's own risk while borrowing strength from the others. Personalization methods differ only in how they couple the shared and the private parts. Write each client objective as a shared term plus a local term,

where \(w\) is the global model and \(\lambda\) sets how hard the fleet pulls the client back. At \(\lambda \to \infty\) you recover the single global model (pure FedAvg); at \(\lambda \to 0\) you recover fully local training with no sharing. Every method below is a way of picking, structuring, or annealing that coupling.

The personalization toolbox

Partial-model personalization splits the network into a shared representation and a private head. The fleet federates the feature extractor (the layers that learn what a "step" or a "heartbeat" looks like in general), while each client keeps a locally trained classifier or a locally estimated normalization layer. FedPer keeps the head private; FedBN keeps each client's batch-normalization statistics local, which is remarkably effective precisely because normalization absorbs the per-device mean and scale shift that causes most of the non-IID pain. This is the cheapest option: the private part is small, it trains in seconds on-device, and it never leaves the device, so it adds no communication cost and improves privacy.

Regularized local training keeps the whole model personal but tethers it to the global one, exactly the \(\lambda\)-penalized objective above. Ditto is the clean instance: run FedAvg to get \(w\), then have each client solve for its own \(\theta_k\) with the proximal pull toward \(w\). The knob \(\lambda\) is interpretable and per-client tunable, which matters when some users have thousands of labeled windows and others have twenty.

Meta-learning reframes the goal: instead of shipping a model to use directly, ship an initialization that adapts to any client in one or two gradient steps. Per-FedAvg applies the MAML idea inside federation, optimizing \(w\) so that \(w - \alpha \nabla F_k(w)\) is good for every \(k\). The fleet learns "a place to start from" and the client finishes the job locally.

Client clustering admits that clients fall into groups (left-handed versus right-handed, phone-in-pocket versus phone-in-hand, one wearable model versus another). IFCA alternates between assigning each client to its best-fitting cluster model and updating each cluster model from its members, yielding a handful of specialists instead of one generalist or thousands of loners.

A minimal personalization head

The code below strips the idea to its core so you can see the mechanism, not the framework. Two clients share a fixed random feature map (standing in for the federated backbone) but have opposite label conventions, the sharpest possible non-IID case. We compare a single global linear head fit on the pooled data against a per-client head fit locally on top of the same shared features. The shared features never move; only the tiny head is personalized.

import numpy as np
rng = np.random.default_rng(0)

def features(X, W):                 # shared "federated" backbone (frozen)
    return np.tanh(X @ W)

d, h, n = 4, 32, 400
W = rng.normal(size=(d, h))         # one representation for the whole fleet
beta = rng.normal(size=d)           # the true concept, shared by all

# Two clients with OPPOSITE label conventions (non-IID sign flip)
def make_client(sign):
    X = rng.normal(size=(n, d))
    y = sign * (X @ beta)           # same task, flipped output convention
    return X, y

clients = [make_client(+1.0), make_client(-1.0)]
Phi = [features(X, W) for X, _ in clients]

# (a) One GLOBAL head fit on pooled features -> forced compromise
Phi_all = np.vstack(Phi)
y_all   = np.concatenate([y for _, y in clients])
w_glob, *_ = np.linalg.lstsq(Phi_all, y_all, rcond=None)

# (b) PERSONALIZED head per client on top of the SAME shared features
def mse(P, y, w): return float(np.mean((P @ w - y) ** 2))
for k, ((X, y), P) in enumerate(zip(clients, Phi)):
    w_loc, *_ = np.linalg.lstsq(P, y, rcond=None)
    print(f"client {k}: global MSE={mse(P, y, w_glob):6.3f}  "
          f"personalized MSE={mse(P, y, w_loc):6.3f}")

Run it and the global head posts a large error for both clients while the personalized heads collapse to near zero. The sign flip is a caricature, but real fleets contain softer versions of it everywhere: a user who labels light jogging as "running," a chest strap mounted upside down, a glucose sensor with a personal calibration offset. The lesson transfers directly.

When to personalize, and the traps

When. Personalize when three things hold: clients are genuinely heterogeneous (if they were IID, the global model would already be optimal and personalization only adds variance); each client has enough local labels to fit the private part without overfitting; and the per-client distribution is stable enough that yesterday's adaptation still helps today. If the client's distribution drifts within a session rather than differing between clients, you want test-time adaptation instead (Chapter 66), which reacts online without labels.

The traps. First, catastrophic forgetting: aggressive local fine-tuning can erase the shared knowledge that made the fleet valuable, so the head that nails this week's activities forgets the rare fall it was meant to catch. The regularization tether \(\lambda\) and the on-device continual-learning defenses of Chapter 62 exist to prevent exactly this. Second, cold start: a brand-new client has no local data, so it must fall back to the global (or its cluster's) model until it earns the right to personalize. Third, and most dangerous, self-flattering evaluation. You must measure personalized accuracy on each client's own held-out future data, never on the data you adapted with, or you report memorization as generalization. Report the per-client accuracy distribution, not just the mean, because personalization that helps the average while hurting a vulnerable tail is a regression dressed as a win.

What's Next

In Section 64.4, we confront the uncomfortable fact that even model updates, not just raw data, can leak the individuals who produced them. We introduce differential privacy to bound that leakage formally and secure aggregation to hide any single client's contribution inside the sum, then measure the utility cost these guarantees impose, a cost that personalization, with its thinly-populated per-client parameters, makes especially sharp.