Safety of self-updating models

"Every night I taught myself something new. Nobody checked my homework. By spring I had quietly become a different model than the one they certified, and no one could say exactly when."

An Unsupervised AI Agent

This section closes Chapter 62 by adding guardrails around every mechanism the earlier sections built: the replay of Section 62.3, the prototype imprinting of Section 62.4, and the personalisation of Section 62.6. It assumes you are comfortable with calibrated confidence and rejection (Chapter 18) and with distribution shift as a first-class problem (Chapter 66). It sets up the fleet-scale and functional-safety treatments of Chapter 69 and Chapter 68.

The threat model: what can go wrong when a model teaches itself

Name the failures before defending against them. On-device self-updating faces four distinct hazards, and they demand different countermeasures. Silent degradation is the base case: honest data, honest labels, but the update overfits a personalisation quirk or amplifies catastrophic forgetting (Section 62.2) so that overall accuracy quietly falls. Label poisoning is adversarial: a wearer, or an attacker with physical access to the sensor, supplies inputs whose accepted labels steer the model, for example registering the attacker's own gait as the authorised user, or teaching a fault detector that a real fault is "normal". Concept-drift capture is subtler: a genuinely shifted but transient input (a temporary sensor fault, an unusual environment) gets learned as if it were the new steady state. Envelope violation is the certification hazard: even a beneficial update moves a model that was validated and, in regulated domains, frozen at a specific version outside the state that a regulator or a functional-safety case signed off on. The unifying observation is that on-device learning removes the human gate that a normal MLOps pipeline (Chapter 69) relies on, so the gate has to be reconstructed in software, on the device, before the weights are allowed to move.

The accept-or-reject gate and the sentinel set

The central mechanism is a validation gate. The device keeps a small, immutable sentinel set: a few dozen to a few hundred labelled examples, curated at build time to cover the safety-critical classes and the base distribution, stored in read-only flash so no field process can alter it. An update is never applied in place. Instead the candidate weights \(\theta'\) are computed off to the side, scored on the sentinel set, and adopted only if they do not regress it. Formally, accept \(\theta'\) over the incumbent \(\theta\) only when

where \(\mathcal{A}_{\text{sent}}\) is accuracy (or a cost-weighted metric that punishes false negatives on critical classes) and \(\varepsilon\) is a small tolerance, often zero for safety-critical classes and slightly positive elsewhere to allow benign noise. The sentinel set is deliberately not the personalisation stream: it is the memory of what the model must never lose. Because it is leakage-safe by construction (curated once, never touched by field data, echoing the split discipline of Chapter 5), a pass on the sentinel is trustworthy evidence that the update did not trade away known competence for a few new samples. To bound the update, clip the parameter delta to a trust region of radius \(\rho\), \(\lVert \theta' - \theta \rVert_2 \le \rho\); to detect poisoning, require that accepted new examples clear a confidence or conformal-rejection threshold (Chapter 18) so that a wildly out-of-distribution "teaching" sample is refused rather than learned.

import numpy as np

def safe_update(theta, learn_step, sentinel_X, sentinel_y,
                rho=0.5, eps=0.0):
    """Gate + bound + rollback around one on-device learning step."""
    base_acc = accuracy(theta, sentinel_X, sentinel_y)   # score incumbent

    theta_new = learn_step(theta)                        # candidate weights

    delta = theta_new - theta                            # bound the change
    norm = np.linalg.norm(delta)
    if norm > rho:                                       # trust region clip
        theta_new = theta + delta * (rho / norm)

    new_acc = accuracy(theta_new, sentinel_X, sentinel_y)
    if new_acc >= base_acc - eps:                        # gate on sentinel
        return theta_new, "accepted"                     # commit new checkpoint
    return theta, "rejected: sentinel regression"        # roll back to known-good

def gate_sample(x, conf, tau=0.9):                       # poisoning guard
    return conf >= tau        # only confidently-labelled samples may teach

Listing 62.7.1 makes the three safety properties concrete in one function: the trust-region clip is the bound, the sentinel comparison is the gate, and returning the old theta on failure is the reversibility. Notice that the learning rule itself, learn_step, is a black box; the safety wrapper does not care whether it is a replay update, a prototype imprint, or a gradient step. That separation is what lets you bolt these guarantees onto any of the earlier sections' methods.

Envelopes, logging, and knowing when to stop learning

Gating each update is necessary but not sufficient, because many small accepted steps can drift a model a long way. Two additional controls matter. First, an update budget and drift monitor: track the cumulative parameter movement and the running sentinel margin, and when the model approaches the boundary of its validated envelope, freeze self-updating and require an authenticated over-the-air model from the fleet operator instead. This is where on-device learning hands back to fleet MLOps (Chapter 69). Second, an audit log: record for every update the timestamp, the number of samples, their gated confidences, the sentinel scores before and after, and the accept/reject decision, so that a field regression is diagnosable after the fact rather than a mystery. In functionally-safe domains the safest choice is often to let the model propose and a human or a fleet server dispose: the device accumulates candidate updates and evidence, but the certified weights change only through the governed channel, keeping the deployed model inside the envelope that ISO 26262 and the SOTIF framing of ISO 21448 (Chapter 68) assume.

What's Next

In Chapter 63, we drop the assumption that the device even has continuous power. When a sensor node harvests its energy and computes in intermittent bursts between brownouts, inference and learning must survive being interrupted mid-step, checkpoint their state, and resume, pushing the safety and continuity ideas of this chapter down to the level of a single capacitor's charge.