Catastrophic forgetting on the edge

"You taught me to recognize your new gym. In exchange, I quietly forgot how to recognize you sleeping. Nobody mentioned this was the deal."

A Plastic AI Agent

This section assumes you are comfortable with gradient descent on a neural network and with the online, single-pass setting introduced in Chapter 60. We lean on the Fisher information matrix from the estimation primer in Chapter 4. The replay and few-shot remedies previewed here are developed fully in Section 62.3 and Section 62.4; our job now is to understand the disease, not yet to cure it.

The mechanism: shared weights and the stability-plasticity dilemma

A neural network stores every task in the same pool of weights. When you fine-tune on task \(B\), the gradient of the task-\(B\) loss knows nothing about task \(A\); it moves each weight in whatever direction lowers the \(B\) loss fastest. If a weight was load-bearing for \(A\), that move degrades \(A\) with no penalty, because \(A\)'s error signal is no longer in the batch. Forgetting is not decay or noise. It is the optimizer doing exactly its job on the wrong objective. Formally, sequential training minimizes \(\mathcal{L}_B(\theta)\) starting from \(\theta_A^\star\), and nothing in that objective keeps \(\theta\) near the region where \(\mathcal{L}_A\) is small.

This is one face of the stability-plasticity dilemma: a system rigid enough to protect old knowledge (stable) cannot easily absorb new knowledge (plastic), and vice versa. Freeze the weights and you never forget but never learn; let them move freely and you learn instantly but forget just as fast. Every continual-learning method is a particular compromise on this axis. The regularization family, exemplified by Elastic Weight Consolidation (EWC), makes the compromise explicit by adding a quadratic penalty that pins important weights in place:

$$ \mathcal{L}(\theta) = \mathcal{L}_B(\theta) + \frac{\lambda}{2}\sum_i F_i \,\big(\theta_i - \theta_{A,i}^\star\big)^2 . $$

Here \(F_i\) is the diagonal of the Fisher information matrix, an estimate of how sharply the old task's loss depends on weight \(i\). Weights with large \(F_i\) are stiff; the rest stay free to learn \(B\). The strength \(\lambda\) is the stability-plasticity knob in numeric form. This is elegant, but note what it costs to store: a full copy of \(\theta_A^\star\) and of \(F\), that is, twice the model's parameters kept in memory purely to remember. On a device with a few hundred kilobytes of RAM that overhead alone can be disqualifying, which is the theme of the next subsection.

Why the edge makes it worse

Catastrophic forgetting is a general problem, but four properties of on-device learning turn a manageable nuisance into a first-order risk. First, no data rehearsal: the classic cloud fix is to interleave old and new data, but on the edge the old data has been discarded for privacy or storage reasons (see the federated motivation in Chapter 64), so the naive defense is simply unavailable. Second, a single ordered pass: the sensor stream is non-IID and arrives in temporal blocks, a user runs for a week, then swims for a week, so the model sees long runs of one distribution and drifts hard toward it, exactly the worst case for interference. Third, a tiny memory and compute budget: EWC's Fisher matrix, a replay buffer, or a second frozen copy of the network may not fit at all, and the energy cost of the extra passes is itself constrained (Section 62.5). Fourth, quantized, low-precision weights: an int8 model has coarse weight steps, so a single update can jump across several quantization levels, making forgetting both more abrupt and harder to regularize smoothly.

The compounding effect is what makes this an advanced, research-frontier topic rather than a solved one. A method that controls forgetting beautifully on a GPU with a 10,000-example replay buffer may be useless when the buffer must shrink to 50 latent vectors and the Fisher matrix must be dropped entirely. The design space is genuinely three-way: forgetting, memory, and energy, and you cannot maximize all three.

Measuring forgetting honestly

You cannot manage what you do not measure, and a single accuracy number hides forgetting completely: a model that has traded all of task \(A\) for task \(B\) can post a fine average on today's data. The standard instrument is the accuracy matrix \(R\), where \(R_{i,j}\) is the accuracy on task \(j\) after the model has finished training through task \(i\). From it we read three quantities. Average accuracy is the mean of the last row, overall competence after all tasks. Backward transfer (BWT) averages \(R_{T,j} - R_{j,j}\) over earlier tasks: how much learning later tasks helped (positive) or hurt (negative) the ones already learned; strongly negative BWT is the fingerprint of forgetting. Forgetting reports, per task, the gap between its best-ever accuracy and its final accuracy. Crucially, all of this must be evaluated on held-out, per-user and per-session splits, the leakage-safe discipline of Chapter 5, or you will measure memorization of the replay buffer instead of retention.

import numpy as np

def continual_metrics(R):
    """R[i, j] = accuracy on task j after training through task i (T x T)."""
    R = np.asarray(R, dtype=float)
    T = R.shape[0]
    final = R[-1]                                   # accuracy on every task at the end
    avg_acc = final.mean()
    bwt = np.mean([R[-1, j] - R[j, j] for j in range(T - 1)])   # < 0 means forgetting
    forgetting = np.mean([R[:, j].max() - R[-1, j] for j in range(T - 1)])
    return {"avg_acc": avg_acc, "bwt": bwt, "forgetting": forgetting}

# A 3-task run: each new task wrecks the previous one (diagonal high, off-diagonal collapses)
R = [[0.92, 0.00, 0.00],
     [0.61, 0.90, 0.00],
     [0.55, 0.58, 0.89]]
print(continual_metrics(R))
# -> {'avg_acc': 0.673, 'bwt': -0.315, 'forgetting': 0.315}

The estimator above is the first thing to wire into any on-device learning experiment, because it converts the vague fear of forgetting into an auditable number you can track across a fleet. Pair it with a calibration check (Chapter 18), since a forgetting model often becomes overconfident on the classes it is losing, and the confidence drift is an early warning that fires before accuracy does.

from avalanche.benchmarks.generators import nc_benchmark
from avalanche.training import EWC
from avalanche.evaluation.metrics import forgetting_metrics, accuracy_metrics

bench = nc_benchmark(train_ds, test_ds, n_experiences=5, task_labels=False)
strategy = EWC(model, optimizer, criterion, ewc_lambda=0.4,
               evaluator=default_evaluator([accuracy_metrics(), forgetting_metrics()]))
for experience in bench.train_stream:
    strategy.train(experience)
    strategy.eval(bench.test_stream)     # full accuracy matrix + forgetting, logged

What's Next

In Section 62.3, we take the most effective family of remedies, rehearsal, and make it fit the device. Storing raw sensor windows is impossible on kilobytes of RAM, so we turn to latent and quantized replay: rehearsing a handful of compressed intermediate activations that let the old-task gradients reappear in every batch at a fraction of the memory cost, the technique that would have saved the rowing watch.