"The population model was ninety percent sure the user was asleep. The user was doing bicep curls with a very slow tempo. One resting heart rate does not fit all wrists."
A Humbled AI Agent
The big picture
A pretrained sensor foundation model arrives knowing the general physics of PPG, accelerometry, and the way physiological channels co-move, learned from millions of hours across a hundred thousand strangers. What it does not know is you: your resting heart rate, your gait signature, the exact way your particular watch sits on your particular wrist. Sensor data has enormous between-subject variance: two people performing the identical activity can produce signal distributions further apart than the same person doing two different activities. A population model that is excellent on average can be mediocre for any single individual. This section is about closing that gap cheaply. We study the adaptation spectrum from a frozen zero-shot encoder to a fully fine-tuned one, the parameter-efficient middle that dominates in practice, per-user personalization from a handful of labels, and how to evaluate all of it without fooling yourself. The theme throughout: adapt the smallest number of parameters that buys the accuracy you need, because labels are scarce, users are many, and the base model is expensive to move.
This section assumes you understand self-supervised pretraining from Chapter 17 and the sensor foundation models built on it earlier in this chapter. It leans on calibration and conformal prediction from Chapter 18, and it is the training-time cousin of the test-time adaptation ideas developed for distribution shift in Chapter 66.
The adaptation spectrum: how much of the model do you touch
What you are choosing among is how many parameters to update for a new task or a new user. At one end, zero-shot reuses the frozen encoder and a task template with no gradient steps. Next, a linear probe freezes the encoder and trains only a linear head on top of the pretrained embeddings. Then parameter-efficient fine-tuning (PEFT) inserts a small number of trainable weights (adapters, LoRA matrices, or learned prefixes) while the backbone stays frozen. At the far end, full fine-tuning updates every weight. Why the choice matters: the amount of trainable capacity should scale with how much labeled data you have and how far the target distribution sits from pretraining. Full fine-tuning on a few hundred labels overfits and, worse, causes catastrophic forgetting of the general prior that made the model useful. A linear probe under-fits when the target task needs features the frozen encoder never learned to expose. When to pick each: linear probe for tens of labels and a target close to pretraining; PEFT for hundreds to a few thousand labels; full fine-tuning only with abundant labels and a genuinely new domain, and even then with a small learning rate on the backbone.
Key insight: personalization is adaptation with a population prior
Per-user personalization is not a different technique from task adaptation; it is the same adaptation run with a tiny dataset (one person) and a strong prior (the population model). The winning move is almost never to fine-tune a fresh per-user model from scratch. It is to start from the population weights and nudge them, or better, to freeze them and learn a few user-specific parameters that shrink toward the population solution when the user has little data. That shrinkage is what prevents a noisy handful of self-labeled samples from destroying a model that already works reasonably well.
Parameter-efficient personalization: LoRA, adapters, and prefixes
How PEFT keeps per-user cost tiny is by never duplicating the backbone. In low-rank adaptation (LoRA), a frozen weight matrix \(W_0 \in \mathbb{R}^{d\times k}\) gets an additive low-rank update so the effective weight is
$$W = W_0 + \Delta W = W_0 + \frac{\alpha}{r}\, B A, \qquad A \in \mathbb{R}^{r\times k},\; B \in \mathbb{R}^{d\times r},\; r \ll \min(d,k).$$Only \(A\) and \(B\) train, so a rank-8 adapter on a large attention projection is thousands of parameters instead of millions. Why this is ideal for a sensor fleet: you ship one frozen backbone to every device and store a small per-user adapter (a few tens of kilobytes) that can live on the device or in the user's account. Swapping users is swapping adapters, not reloading a model. Adapters (small bottleneck MLPs inserted between layers) and prefix or prompt tuning (a handful of learned tokens prepended to the sequence) offer the same bargain with different inductive biases. For the token-grid sensor models of this chapter, prefix tokens are attractive because they inject user context without touching the convolutional or attention weights at all, and they compose naturally with the multi-device inputs of Chapter 26's activity recognition setting.
Library shortcut: PEFT instead of hand-wired adapters
Wiring LoRA by hand means subclassing every linear layer, managing which parameters require gradients, and writing merge logic for inference, roughly 120 to 200 lines and easy to get subtly wrong. Hugging Face peft reduces it to about 5 lines: wrap the frozen model with get_peft_model(model, LoraConfig(r=8, target_modules=[...])) and train as usual; it freezes the backbone, injects the adapters, and gives you save_pretrained that writes only the tiny per-user delta. The sensor-specific work you keep is choosing which modules to adapt and building the leakage-safe per-user split below.
Few-shot personalization from a frozen encoder
When a user can supply only a few labeled windows (they tapped "this was a run", "this was sleep"), the cheapest reliable personalization skips gradients entirely. What you do is embed the user's few labeled windows with the frozen encoder, form a per-class prototype (the mean embedding), and classify new windows by nearest prototype. How you make it robust with so little data is regularized shrinkage: blend each per-user linear head toward the population head, with the blend weight controlled by how many samples the user gave. The code below fits exactly such a head on frozen embeddings and shrinks it toward the population weights, so a user with two examples inherits almost the population model while a user with two hundred earns a strongly personalized one.
import numpy as np
def personalize_head(Z_user, y_user, W_pop, num_classes, lam=5.0):
"""Ridge-regularized per-user linear head that shrinks toward the
population head W_pop as the user's label count shrinks.
Z_user: (n, d) frozen FM embeddings for this user's labeled windows.
y_user: (n,) integer class labels. W_pop: (num_classes, d) population head.
lam: strength of the pull toward W_pop (acts like a per-user prior)."""
n, d = Z_user.shape
Y = np.eye(num_classes)[y_user] # (n, C) one-hot targets
# Solve (Z^T Z + lam I) W^T = Z^T Y + lam W_pop^T -> personalized W
A = Z_user.T @ Z_user + lam * np.eye(d)
b = Z_user.T @ Y + lam * W_pop.T
W_user = np.linalg.solve(A, b).T # (C, d)
return W_user
rng = np.random.default_rng(0)
d, C = 128, 4
W_pop = rng.normal(scale=0.1, size=(C, d)) # frozen population head
for n in (2, 20, 200): # user gives few -> many labels
Z = rng.normal(size=(n, d)); y = rng.integers(0, C, n)
W_user = personalize_head(Z, y, W_pop, C)
drift = np.linalg.norm(W_user - W_pop) / np.linalg.norm(W_pop)
print(f"n={n:>3} labels -> relative drift from population head: {drift:.2f}")lam fixed, more user labels overcome the pull toward W_pop, so the relative drift grows with n: two labels barely move off the population solution, two hundred earn a strongly personalized head. The frozen encoder that produced Z_user is never updated, so nothing is forgotten.The printed drift climbing with the label count is the shrinkage working: personalization strength is earned by evidence, not assumed. This is the safe default for on-device use because it needs no backpropagation through the backbone, which connects directly to the continual, on-device personalization of Chapter 62 and the privacy-preserving fleet learning of Chapter 64.
Research frontier: test-time and self-supervised personalization
As of 2025 the frontier is personalizing without labels. Test-time training adapts the model on each user's unlabeled stream by re-running the pretraining self-supervised loss (masked reconstruction, contrastive matching) at inference, so the backbone drifts toward the user's distribution before it predicts. Apple's and Google's wearable-FM lines report that a short self-supervised adaptation pass on a user's own recent data recovers much of the gap to a fully supervised per-user model, using zero new labels. The open problems are stability (avoiding drift into degenerate solutions on a single user's narrow data), knowing when a user has shifted enough to warrant re-adaptation, and doing it within a wearable's power budget. These threads run straight into Chapter 66's test-time adaptation.
Practical example: a continuous glucose model that meets a new wearer
A clinical wearables team deploys a pretrained multi-sensor model that estimates glycemic excursions from accelerometry, heart rate, and skin temperature. Out of the box it is well calibrated on the population but systematically over-predicts for wearers with unusually low resting heart rate, a group under-represented in pretraining. Rather than collect a labeled dataset per patient (impractical and clinically burdensome), they attach a rank-4 LoRA adapter to the encoder's attention projections and fit a shrinkage head on the first two weeks of each patient's data, using the sparse fingerstick references the patient already logs. The adapter is 30 KB per patient and lives beside their record. Personalized error drops by roughly a third for the low-heart-rate cohort, the population backbone is never modified so no other patient is affected, and because the head shrinks toward the population solution, a patient who logs almost nothing simply keeps the safe population behavior. Calibration is re-checked per patient with the conformal machinery of Chapter 18 before any personalized number reaches a clinician.
Evaluating personalization without fooling yourself
Why this deserves its own section: personalization is where leakage is easiest and most tempting. If you split windows randomly, a user's morning run lands in train and their afternoon run in test, and the model scores brilliantly by memorizing the person, telling you nothing about generalization to new users. How to evaluate honestly follows Chapter 5: partition by subject, never by window. Report two distinct numbers. First, cross-user generalization: train and personalize on some users, test on held-out users, which measures whether the base model plus adaptation recipe transfers at all. Second, within-user gain: for each held-out user, reserve their earliest data for personalization and their later data (chronologically after) for scoring, which measures what personalization actually buys a real person over time while respecting causality. A method that improves the first number is a better foundation model; a method that improves the second is a better personalization scheme, and you want to know which you built.
Exercise: the personalization gain curve
Take a pretrained sensor encoder and a HAR dataset with per-subject identifiers. Hold out ten subjects entirely. For each held-out subject, use their first \(k\) labeled windows to fit the shrinkage head above (or a rank-4 LoRA adapter) and score on their remaining, chronologically later windows. Sweep \(k \in \{0, 5, 20, 100\}\), where \(k=0\) is the frozen population head. Plot mean per-subject accuracy against \(k\), with error bars across subjects. Then repeat with a random (leaky) window split instead of the chronological per-subject split. Report both curves and explain, in terms of what each split lets the model memorize, why the leaky curve is optimistic and which number you would put in a product spec.
Self-check
- Order zero-shot, linear probe, PEFT, and full fine-tuning by trainable parameter count, and state the label regime in which each is the right default.
- In the shrinkage head, what happens to the personalized weights as
lamgrows very large, and as the user's label count grows very large? Why is this the behavior you want for a cold-start user? - Why does a random window split inflate personalization metrics, and what specific split removes that inflation while still measuring within-user gain?
What's Next
In Section 20.7, we take the adapters, shrinkage heads, and test-time updates from this section and confront them with the hardware: how much compute a wearable can spare for on-device personalization, what a per-user adapter costs in memory and energy, and how privacy constraints reshape where adaptation is allowed to happen at all.