"They asked why I retrained the fleet nine times last month. I said the drift alarm told me to. They asked why the drift alarm fired nine times. I said because I retrained the fleet nine times."
A Self-Reinforcing AI Agent
Prerequisites
This section assumes you already have the monitoring signals from earlier in this chapter: the drift and data-quality detectors of Section 69.3 and the proxy metrics that stand in for delayed labels in Section 69.4. It builds on the versioning and staged-deployment mechanics of Section 69.2, which we treat here as a primitive we call rather than re-explain. It leans on distribution-shift concepts from Chapter 66 and on leakage-safe evaluation from Chapter 65. Our subject is narrower and operational: given those signals, when do you spend the cost of retraining, and how do you push a new model across a heterogeneous fleet without breaking half of it.
The Big Picture
A deployed sensor model decays because the physical world moves underneath it: seasons change, hardware ages, new device revisions ship, users behave in ways the training set never saw. Retraining is the remedy, but retraining is not free and not always safe. Each cycle costs compute, labeling, evaluation, and the operational risk of shipping a regression to thousands of devices at once. A retraining trigger is the decision rule that converts monitoring signals into the binary act of spending that cost. Fleet management is the discipline of applying the result to a population of devices that are not identical, staged so that a bad model is caught on ten devices instead of ten thousand. This section is about designing triggers that fire when they should and only when they should, and rollouts that fail small.
Four families of trigger, and why calendar alone is not enough
The simplest trigger is the scheduled one: retrain every quarter regardless of state. It is trivial to implement and easy to reason about, and for slowly drifting domains it is often correct. Its weakness is that it is blind. It retrains when nothing has changed (wasting compute and injecting avoidable deployment risk) and it waits patiently through a fault that appeared the day after the last cycle. Schedule is a floor, not a strategy.
The second family is performance-triggered: retrain when a measured quality metric crosses a threshold. When ground truth is prompt this is the gold standard, because it responds to the thing you actually care about. In sensor fleets truth is usually late or absent, so in practice you trigger on the proxy metrics of Section 69.4, calibrated confidence, prediction-interval width, agreement between redundant channels, and accept that you are firing on a correlate of accuracy rather than accuracy itself.
The third family is data-triggered: retrain when the input distribution shifts, using the drift statistics of Section 69.3, or when enough new labeled examples have accumulated to move the decision boundary, or when a previously unseen operating regime appears (a new firmware version, a new mounting site, a device revision). The fourth family is event-triggered: a fielded incident, a recall, a regulatory change, or a spoofing discovery (Chapter 68) forces a cycle out of band. A mature fleet wires all four into one controller: schedule sets the cadence floor, proxy and drift raise the priority, and events preempt everything.
Key Insight
A raw threshold is a trap. Sensor metrics are noisy, so a single-sided threshold with no memory will chatter: it crosses, you retrain, the metric recovers slightly, it crosses again, and you have built a machine that retrains in response to its own retraining. Every production trigger needs three things the naive version lacks: hysteresis (fire on a high threshold, only re-arm below a lower one, so noise near the boundary cannot oscillate the decision), persistence (require the condition to hold across a window, not a single sample, to reject spikes), and a cooldown (a minimum interval between cycles so a legitimate but transient shift cannot spend your entire compute budget in a day). The trigger is not the detector; it is the debounced, rate-limited governor sitting on top of the detector.
A debounced multi-signal trigger controller
The controller below fuses several monitoring signals into one retraining decision with the three safeguards from the insight above. Each signal reports a normalized severity in \([0, 1]\); the controller combines them, requires persistence across a window, respects a cooldown, and uses separate arm and disarm thresholds. Note that an event signal bypasses persistence and cooldown, because a confirmed field incident should never be debounced away. The fused severity is a weighted maximum rather than a mean, because for retraining you care about the worst live problem, not the average health.
from dataclasses import dataclass, field
from collections import deque
@dataclass
class RetrainTrigger:
arm: float = 0.70 # fire when fused severity rises above this
disarm: float = 0.45 # only re-arm once it falls back below this (hysteresis)
persist: int = 6 # consecutive windows the condition must hold
cooldown: int = 30 # minimum windows between two retraining cycles
weights: dict = field(default_factory=lambda: {
"drift": 1.0, "proxy_conf": 1.0, "new_regime": 0.8, "label_volume": 0.6})
_armed: bool = True
_hot: deque = field(default_factory=lambda: deque(maxlen=6))
_since_last: int = 999
def step(self, signals: dict, event: bool = False) -> bool:
self._since_last += 1
if event: # incident preempts everything
self._reset()
return True
fused = max(self.weights.get(k, 0.0) * v for k, v in signals.items())
self._hot.append(fused >= self.arm)
persistent = len(self._hot) == self._hot.maxlen and all(self._hot)
if self._armed and persistent and self._since_last >= self.cooldown:
self._reset()
return True
if not self._armed and fused < self.disarm: # re-arm only after cooling off
self._armed = True
return False
def _reset(self):
self._armed, self._since_last = False, 0
self._hot.clear()
# One monitoring tick: drift is high but confidence is still fine.
trig = RetrainTrigger()
fire = trig.step({"drift": 0.82, "proxy_conf": 0.30, "new_regime": 0.0})
print("retrain now?", fire) # False: needs to persist across the window first
event path bypasses all debouncing so a confirmed incident triggers immediately. Signals are the normalized outputs of the detectors from Sections 69.3 and 69.4; this controller only decides whether their combination warrants spending a retraining cycle.The controller above is deliberately small so the policy is auditable: an on-call engineer can read why a cycle fired from the signal trace and the threshold constants. In production you would log the fused severity and per-signal contributions on every tick, so a post-hoc review can answer "why did we retrain on the 14th" without rerunning anything.
Right Tool: don't hand-roll the drift half
The controller above assumes the signals dict already exists. Producing the drift component from raw batches, reference-window bookkeeping, population-stability index or Kolmogorov-Smirnov tests, per-feature aggregation, is roughly 120 to 200 lines to do carefully. Libraries such as river (ADWIN, Page-Hinkley, KSWIN drift detectors) or evidently (dataset-level drift reports) collapse that to about 5 lines: instantiate a detector, call update(x) per sample, read its drift_detected flag straight into signals["drift"]. They handle the sliding windows, the statistical tests, and the multiple-comparison bookkeeping. Keep your own thin trigger policy on top; delegate the detector plumbing.
From "retrain" to "safely deployed": the gated fleet rollout
A fired trigger starts a pipeline, it does not ship a model. The retrained candidate must clear a promotion gate before any device sees it: it is evaluated on a frozen, leakage-safe holdout (the discipline of Chapter 65), and it must beat the incumbent on the primary metric while not regressing on any guardrail slice. The subtle failure here is unique to fleets: a new model can raise fleet-average accuracy while badly regressing a minority cohort, an older device revision, a rare mounting geometry, a specific user population. The gate must therefore be per-cohort, not just global; a candidate that improves the mean but breaks the 2019-hardware cohort should be blocked, not promoted.
Past the gate, you never flip the whole fleet at once. The house pattern is a staged rollout: a shadow phase where the candidate runs alongside the incumbent and its outputs are logged but not acted on; a canary phase on a small, representative slice (often one to five percent, chosen to cover the cohorts, not just random devices); then widening waves with an automatic rollback if any wave's live proxy metrics degrade past a bound. Because ground truth is delayed, the canary's kill switch keys off the same proxy and drift signals that drive the trigger, which is why those signals must be trustworthy before you automate any of this.
Step-Through: the fitness wearable and the winter cohort
A wearable vendor runs a heart-rate and activity model on eight device revisions across roughly four million wrists. In late autumn the drift signal climbs steadily; the trigger's persistence window fills and, past cooldown, it fires. The retrained candidate lifts fleet-average step-detection F1 by 1.4 points and clears the global gate. But the per-cohort gate flags a regression: on the two oldest optical-sensor revisions, worn under thick winter sleeves, recall drops 6 points, because the new training mix under-represented low-perfusion cold-skin signals (the PPG physics of Chapter 30). A naive fleet-average gate would have shipped the regression to millions. Instead the candidate is held, the winter cold-skin cohort is upsampled, and the next candidate clears both gates. It then rolls out shadow first, then a 2 percent canary weighted toward those old revisions, then three widening waves, with proxy-confidence watched per cohort at each step. The regression that a mean would have hidden was caught at the gate; the staged rollout meant that even an escaped fault would have touched a canary, not the fleet.
Fleet structure: one model, many, or a shared backbone
Heterogeneity forces a design choice about how many models the fleet carries. A single global model is cheapest to operate and monitor but must generalize across every device revision and environment, and it is the one most likely to hide cohort regressions. Per-cohort models (one per device generation, region, or use profile) fit each population better but multiply your retraining, versioning, and monitoring surface by the number of cohorts, and they fragment the data so each model sees less. The common middle path is a shared backbone with light per-cohort heads or adapters: the representation is trained fleet-wide, and only a small, cheap head is retrained per cohort when its trigger fires. This is exactly the on-device continual-learning and adapter machinery of Chapter 62, applied at fleet granularity, and it lets a single cohort's trigger fire without forcing a global retrain.
Whatever the structure, retraining introduces its own hazard: a candidate trained on the newest data can suffer catastrophic forgetting of a rare-but-critical regime that the fresh window under-samples. Guard it by keeping a frozen regression suite of hard historical cases (past incidents, rare failure signatures, known-difficult cohorts) that every candidate must still pass, independent of what the recent data looks like. The regression suite is the institutional memory that stops each retrain from quietly relearning last year's mistakes.
Research Frontier
Deciding when to retrain is increasingly framed as a cost-aware sequential decision rather than a fixed threshold. Recent work on data-drift-aware retraining policies and reinforcement-learning schedulers (for example, the "Matchmaker" and cost-aware retraining lines) treats each cycle as an action with a compute-and-risk cost and a stochastic accuracy payoff, and learns a policy that beats both fixed schedules and simple drift thresholds on total cost per unit accuracy. On the fleet side, coupling retraining triggers with test-time adaptation (Chapter 66) is an active area: cheap on-device adaptation can buy time and postpone a full retrain, changing the economics of the trigger itself. Both directions are early, and neither yet has a settled evaluation protocol for delayed-label sensor fleets.
Exercise
Take the RetrainTrigger above and feed it a synthetic 300-tick trace in which drift is a noisy signal that hovers near 0.68 for 40 ticks, then jumps to a sustained 0.9. (1) With the default constants, on which tick does the first cycle fire, and why does the noisy plateau not fire it? (2) Now set persist=1 and cooldown=1 and count how many cycles fire across the trace; explain the chattering. (3) Add a second signal proxy_conf that is 0.0 until tick 200 then jumps to 0.95, and describe how the weighted-max fusion changes the fire schedule versus using a mean.
Self-Check
- Why must a retraining trigger use separate arm and disarm thresholds rather than a single threshold, and what specific failure does the gap prevent?
- A candidate raises fleet-average accuracy by 2 points but you refuse to promote it. Give a concrete, sensor-specific reason a per-cohort gate would block it that a global gate would miss.
- Why does the
eventpath in the controller deliberately skip the persistence window and cooldown, and what would go wrong if it did not?
What's Next
Triggers and rollouts assume you can log the fleet's signals in the first place, but sensor logs are among the most privacy-sensitive data an organization holds: raw waveforms can re-identify a body, a home, or a location. In Section 69.6, we turn to logging under privacy constraints, how to capture enough signal to drive the monitors and triggers of this chapter while minimizing, aggregating, and protecting what leaves the device.