Differential privacy and secure aggregation

"Keeping the raw data on the phone feels safe until you remember the gradient is a lossy photograph of it."

A Prudent AI Agent

This section assumes you have met the FedAvg training loop and the non-IID reality of sensor fleets from Section 64.2, and the motivation for keeping raw traces on-device from Section 64.1. The probability and estimation tools behind the noise mechanisms come from Chapter 4, and the biometric-privacy stakes that make all of this matter are laid out in Chapter 34.

Why a gradient leaks

What is the threat? Federated updates are functions of private data, and functions leak. Concretely, gradient-inversion attacks reconstruct input windows from a shared gradient by optimizing a dummy input until its gradient matches the observed one. On sensor data this is worse than on images: an accelerometer or PPG window is low-dimensional and smooth, so the reconstruction is often near-perfect. Membership-inference attacks ask a weaker but still damaging question, "was this specific person's data used?", and answer it from the model's confidence. A hospital that federates an ECG classifier across clinics may inadvertently let a curious clinic confirm that a named patient was in another clinic's cohort.

Why does keeping data local not fix this? Because the object you release, the update, is a compressed but information-rich summary of the batch that produced it. Locality controls where computation happens; it does not bound how much the output reveals. That bound is exactly what differential privacy provides.

Differential privacy: the contract and the dial

What is DP? A randomized mechanism \(\mathcal{M}\) is \((\varepsilon, \delta)\)-differentially private if for any two datasets \(D, D'\) differing in one contributor's data, and any set of outcomes \(S\),

Read it as a contract: whether or not you are in the dataset, the distribution of published models is nearly identical, so no adversary, however clever or well-resourced later, can tell from the output that you participated. \(\varepsilon\) is the privacy budget (smaller is stronger; single-digit values are the usual target), and \(\delta\) is a small failure probability, conventionally set below \(1/N\) for a fleet of \(N\) users.

How do we achieve it in federated training? The workhorse is DP-SGD with two steps per update: clip each per-example (or per-user) gradient to a fixed \(\ell_2\) norm \(C\) so no single contributor can dominate, then add Gaussian noise calibrated to that sensitivity. The clipped, noised update is

The crucial design choice for sensor fleets is the unit of privacy. Protecting one training example is nearly useless when a wearable contributes thousands of correlated windows from the same person; you want user-level DP, where the neighboring datasets differ by removing all of one user's data. That is why federated DP clips and noises at the level of each client's total update, not each sample. A privacy accountant (Renyi DP or the numerically tighter PLD accountant) then tracks the cumulative \(\varepsilon\) as rounds accumulate, because privacy loss composes across every round a user is sampled into.

Secure aggregation: summing without seeing

What is SecAgg? A cryptographic protocol that computes \(\sum_i u_i\) over client updates \(u_i\) such that the server learns only the sum, never any individual \(u_i\). How: each pair of clients agrees on a shared random mask (via Diffie-Hellman key exchange); client \(i\) adds masks for peers with higher IDs and subtracts masks for peers with lower IDs. When all updates are summed, every pairwise mask cancels exactly, leaving the clean total. Secret-sharing of the mask seeds lets the server recover the sum even when some phones drop out mid-round, a routine event for battery-powered sensors, without ever unmasking a survivor.

Why does this matter beyond DP? Because the honest-but-curious server is the most realistic adversary in a commercial fleet. SecAgg removes the individual update from the server's view entirely, so a subpoena, a breach, or an over-logging bug on the server exposes only aggregates. When to reach for it: any cross-device setting with an untrusted or semi-trusted coordinator, especially health and biometric data. Its cost is communication and a dropout-tolerance ceiling; classic SecAgg scales roughly quadratically in per-client cost with cohort size, so large cohorts use SecAgg+ or ring-based variants.

Reading the utility cost, and tuning it

Privacy is never free, and the bill lands hardest exactly where sensor fleets are most heterogeneous. Gradient clipping disproportionately shrinks the large, informative updates from rare users (the night owls, the atypical gaits), so DP amplifies the fairness gap that non-IID data already creates. Three levers recover most of the lost utility: a larger cohort per round (privacy amplification by subsampling means more clients lets you add proportionally less noise for the same \(\varepsilon\)); a well-chosen clip norm (set \(C\) near the median un-clipped update norm, and consider adaptive clipping that tracks it); and fewer, higher-quality rounds, since every round a user participates in spends budget. The short program below shows the two DP-SGD operations in isolation so the mechanism is concrete before a library hides it.

import numpy as np

def privatize_updates(client_updates, clip_norm=1.0, noise_mult=1.1):
    """One user-level DP aggregation step: clip each client, sum, add noise."""
    clipped = []
    for u in client_updates:                    # u = one client's flat update vector
        norm = np.linalg.norm(u)
        clipped.append(u * min(1.0, clip_norm / (norm + 1e-12)))
    summed = np.sum(clipped, axis=0)
    sigma = noise_mult * clip_norm              # Gaussian std scales with sensitivity
    noise = np.random.normal(0.0, sigma, size=summed.shape)
    return (summed + noise) / len(client_updates)   # noised average update

rng = np.random.default_rng(0)
cohort = [rng.normal(0, s, size=8) for s in (0.3, 0.4, 2.5, 0.35)]  # one outlier user
private_avg = privatize_updates(cohort, clip_norm=1.0, noise_mult=1.1)
print(np.round(private_avg, 3))

What's Next

In Section 64.5, we turn from protecting the update to partitioning the model itself: split and split-federated learning cut the network between device and server so that neither holds the whole picture, trading a different slice of privacy, compute, and communication than the mechanisms in this section.