"You asked me to backpropagate on a coin-cell microcontroller, with a batch of one, no labels, and a new user every morning. I declined, and adapted anyway."
A Gradient-Averse AI Agent
Prerequisites
This section builds directly on the gradient-based test-time adaptation of Section 66.4 (TENT, CoTTA) and defines itself against it. It assumes the human-activity recognition pipeline and the subject-shift problem of Chapter 26, the softmax-confidence and entropy tools of Chapter 18, and the frozen-feature-extractor mindset of Chapter 17. The deployment target throughout is the resource-constrained edge of Chapter 59, where an autograd engine is a luxury you may not have.
The Big Picture
A wrist-worn activity classifier trained on 40 volunteers meets user 41, whose stride, wrist angle, and pace it has never seen. Accuracy on "walking" versus "stairs" quietly drops ten points. The obvious fix from Section 66.4 is to backpropagate an entropy loss on the incoming stream. But on a wearable that means shipping an optimizer, choosing a learning rate that no one will tune per-user, adapting on batches of one, and risking the silent collapse where TENT drives every window to a single confident class. Optimization-free test-time adaptation refuses the gradient entirely. It keeps the network weights frozen and instead corrects the outputs or the decision boundary in closed form, using only the geometry of the incoming features. No learning rate, no optimizer state, no forgetting, and a per-window cost measured in matrix multiplies. This section shows why that trade is often the right one for cross-person HAR, and exactly how three gradient-free methods work.
Why refuse the gradient at all
Gradient-based TTA earns its accuracy, but every term it adds is a liability on a wearable. It needs a learning rate, and the field-honest answer is that you cannot re-tune it for each new wrist; a rate that recovers user 41 can destabilize user 42. It adapts on whatever batch the device can buffer, and cross-person HAR streams arrive one 2-second window at a time, so the batch statistics that TENT leans on are noisy or degenerate. Worst, unsupervised entropy minimization has a trivial optimum: predict one class with full confidence for everything. Under the sustained single-activity stretches typical of daily wear (an hour of sitting), that collapse is not hypothetical. And all of this presumes an autograd runtime and the memory for activations and optimizer moments, which a Chapter 59 microcontroller does not have.
Optimization-free methods sidestep the whole list. Their premise is that the frozen backbone, trained on 40 subjects, already produces usable features for subject 41; what shifts is not the feature extractor's competence but the mapping from features to labels, plus the class balance the user happens to exhibit. Fixing that mapping is a small, convex, closed-form problem. The three families below attack it at three different depths of the network.
Key Insight
Gradient-based TTA changes the model; optimization-free TTA changes the answer. Because the weights never move, there is no catastrophic forgetting, no learning-rate hyperparameter, and the adaptation is fully reversible: run the backbone once, then post-process its logits or its penultimate features. This reframes adaptation from "continue training in the field" to "solve a tiny inference-time correction," which is why it fits an edge budget that gradient descent never will.
Three gradient-free families, from shallow to deep
The methods differ in where they intervene. Statistic recalibration is the shallowest: replace the batch-normalization running mean and variance (frozen from training) with statistics computed on the incoming test window, a pure forward-pass operation. This is the optimization-free core that TENT wraps a gradient step around; used alone it already recovers a surprising fraction of the drop from covariate shift, because much of cross-person variation is a shift in feature mean and scale. Template (prototype) adjustment intervenes at the classifier: T3A (Test-Time Template Adjustment) discards the trained linear head's fixed weight vectors and rebuilds each class centroid from confidently-predicted test features, so the decision boundary migrates toward the new user's actual feature cloud without a single gradient. Output correction is the deepest in reasoning while touching the network least: LAME (Laplacian Adjusted Maximum-likelihood Estimation) leaves logits alone but re-solves the soft label assignment so that windows that are neighbors in feature space receive similar predictions, encoding the prior that consecutive HAR windows rarely jump between unrelated activities.
Research Frontier
T3A (Iwasawa and Matsuo, NeurIPS 2021) and LAME (Boudiaf et al., CVPR 2022) established that a backprop-free correction can match or beat TENT-style adaptation while being provably immune to the entropy-collapse failure mode, because neither has a degenerate all-one-class optimum. LAME in particular is designed for the non-episodic, single-sample-friendly regime that streaming HAR lives in, and its authors show it degrades gracefully where gradient methods diverge. The open frontier is combining these output-space corrections with the conformal risk control of Section 66.6 so that an unlabeled adaptation still carries an honest guarantee.
LAME in detail
LAME makes the neighbor prior precise. Let the frozen backbone produce, for a buffer of \(N\) windows, softmax probabilities \(p_i \in \Delta^{K}\) and \(\ell_2\)-normalized penultimate features \(f_i\). We seek adjusted soft labels \(z_i \in \Delta^{K}\) that stay close to the network's beliefs yet vary smoothly over the feature-space graph. With an affinity \(w_{ij}\) (a k-nearest-neighbor kernel on the \(f_i\)), LAME minimizes
$$ \mathcal{L}(Z) = -\sum_{i=1}^{N} z_i^\top \log p_i \;-\; \sum_{i,j} w_{ij}\, z_i^\top z_j, $$where the first term pulls each assignment toward the backbone's evidence and the second rewards agreement between neighbors. The objective is concave in each \(z_i\) over the simplex, and the fixed-point update that solves it is simply
\[ z_i \;\leftarrow\; \mathrm{softmax}\!\Big(\log p_i \;+\; \sum_{j} w_{ij}\, z_j\Big), \]iterated to convergence (typically under ten passes). Nothing here touches the weights; the entire cost is building a small affinity matrix and a handful of sparse matrix-vector products. Listing 66.5.1 implements the whole method in a dozen lines, and the crucial property to notice is that the uniform assignment is not a fixed point unless the logits already say so, which is exactly why LAME cannot collapse the way entropy minimization can.
import numpy as np
def lame(features, probs, k=5, iters=10):
# features: (N, D) L2-normalized penultimate features from a FROZEN backbone
# probs: (N, K) softmax outputs; weights are never updated
N = features.shape[0]
S = features @ features.T # cosine affinity (features are unit norm)
knn = np.argsort(-S, axis=1)[:, 1:k + 1] # k nearest neighbors, excluding self
W = np.zeros((N, N))
for i in range(N):
W[i, knn[i]] = S[i, knn[i]]
W = 0.5 * (W + W.T) # symmetrize the affinity graph
logp = np.log(np.clip(probs, 1e-8, 1.0))
Z = probs.copy()
for _ in range(iters): # closed-form fixed point, no gradients
A = logp + W @ Z
A -= A.max(axis=1, keepdims=True)
E = np.exp(A)
Z = E / E.sum(axis=1, keepdims=True) # project each row onto the simplex
return Z # adjusted soft labels; argmax to decidefeatures and probs; adaptation is a k-NN affinity graph plus a few smoothing iterations. There is no optimizer, no learning rate, and no weight update, so the method cannot forget its training or collapse to a single class.In Practice: a rehab wearable meeting patient number one
A clinical gait-monitoring band, trained on healthy volunteers, is deployed to a stroke-rehabilitation ward where every patient walks with an asymmetric, slow, hemiparetic gait the training set never contained. On the first day the frozen model confuses "slow shuffling walk" with "standing," because the acceleration envelope is muted. TENT was ruled out: the ward's safety review would not sign off on a model whose weights silently mutate on a medical device, and a batch of one produced unstable normalization. LAME, applied over a rolling 64-window buffer, recovered eight of the eleven lost accuracy points on the first patient with zero weight changes and a per-window budget the band's Chapter 59 processor absorbed without waking a second core. Because the correction was output-only and reversible, the ward kept the certified backbone bit-for-bit and logged the adjustment separately for audit. Confidence tracked the change honestly (Chapter 18), rising only where neighbors agreed.
When it works, and when it does not
Optimization-free TTA rests on one load-bearing assumption: the frozen features are still discriminative for the new user, even if the head is miscalibrated for them. When that holds (mild-to-moderate covariate shift, a strong self-supervised backbone from Chapter 17), rebuilding prototypes or smoothing outputs recovers most of the loss for almost no compute. When the backbone's features themselves degrade (a wholly new sensor placement, a device-firmware change that alters the sampling geometry of Chapter 3), no output correction can invent information the features lost, and only genuine feature-space adaptation or retraining will help. LAME carries a second, subtler caveat: its neighbor prior assumes clustered class structure, so on a test buffer that is almost entirely one activity it can over-smooth minority windows into the majority. The practical rule is to gate it on a buffer with adequate predicted-class diversity, and to monitor the adaptation's risk rather than trust it blindly, which is precisely the subject of the next section.
The Right Tool
Re-implementing the full optimization-free family (BN-statistic recalibration hooks, a T3A support-set manager with entropy-based pruning, and a LAME solver with several affinity kernels) from scratch runs to roughly 250 lines of careful bookkeeping. Modern TTA libraries expose each as a one-line wrapper around your frozen model:
from tta_library import LAME, T3A # e.g. a robustbench-style TTA API
adapted = LAME(frozen_model, affinity="knn", knn=5)
preds = adapted(test_batch) # forward-only; weights stay frozenListing 66.5.2 hides the plumbing, but the assumption analysis above is still your job: a library will happily run LAME on a single-activity buffer and quietly over-smooth it.
Exercise
Take a HAR model trained with a leave-subjects-out split from Chapter 26 and evaluate it on held-out subjects to establish the cross-person drop. Now apply, in turn, (a) test-time BN-statistic recalibration only, (b) T3A prototype adjustment, and (c) LAME from Listing 66.5.1. Report the accuracy recovered by each, the per-window latency, and, critically, construct a mostly-single-activity buffer and show whether LAME over-smooths its minority windows. Which method gives the best recovery-per-millisecond on your target device?
Self-Check
1. Entropy minimization (TENT) has a trivial degenerate optimum. State it, and explain why LAME's fixed point does not share it.
2. Optimization-free TTA freezes the backbone. What single assumption about the frozen features must hold for output correction to help, and name a shift that violates it.
3. Why is a batch of one, common in streaming HAR, a bigger problem for gradient-based TTA than for LAME or T3A?
What's Next
In Section 66.6, we confront the uncomfortable truth that every method in this chapter adapts on unlabeled data, so it can silently make things worse. We will build the monitoring that watches an adaptation in flight, detects when the recovery has turned into drift, and attaches an honest risk bound to a correction that was made without a single ground-truth label.