"On the cloud they measure my learning in dollars per GPU-hour. Here, on a coin cell, I measure it in days of life I will never get back. Every gradient I take is a gradient stolen from tomorrow's inference."
A Thrifty AI Agent
The Big Picture
Every continual-learning method in this chapter, replay in Section 62.3, prototype imprinting in Section 62.4, assumes the device can afford to run the update. On a wall-powered server that assumption is free. On a battery, it is the whole design. Training draws two to five times the current of inference for the same window, writing an adapted weight to flash costs orders of magnitude more energy than reading it, and a device that trains too eagerly simply dies before its warranty. This section treats energy as a hard budget, not an afterthought. We build a joule ledger for a learning device, show why memory traffic and not multiply-accumulates usually dominates the bill, and turn "should the model adapt right now?" into an arithmetic decision: does the expected accuracy gain justify the charge it will burn? Get this wrong and on-device learning is a liability; get it right and a model improves itself for months on a cell you never replace.
This section assumes you can read a model's compute cost in multiply-accumulate operations and understand quantisation, from Chapter 59, and the microcontroller memory hierarchy (SRAM, flash, external DRAM) from Chapter 61. Where energy comes from a harvester rather than a fixed cell, the accounting connects directly to Chapter 63.
Why a gradient step costs far more than a forward pass
Start with the compute asymmetry. Backpropagation runs the forward pass, then a backward pass of comparable arithmetic, then a weight-update pass. A rule of thumb that holds across most architectures is that a full training step costs roughly \(2\times\) to \(3\times\) the multiply-accumulates of inference on the same input. But arithmetic is the cheap part. Training also needs to keep every layer's activations in memory to compute gradients, so peak RAM balloons from the single largest activation (inference) to the sum of all activations (training), often \(10\times\) larger. On an embedded part that difference decides whether the update fits in SRAM at all or spills to external DRAM, and every off-chip access is where the joules hide.
Model the energy of one learn step as the sum of a compute term and a memory term:
\[ E_{\text{step}} = N_{\text{mac}}\, e_{\text{mac}} \;+\; \sum_{k} B_k\, e_{\text{mem},k}, \]where \(N_{\text{mac}}\) is the operation count, \(e_{\text{mac}}\) the energy per multiply-accumulate, \(B_k\) the bytes moved at memory level \(k\), and \(e_{\text{mem},k}\) its per-byte energy. The numbers that matter are the ratios. On a modern low-power process, a 32-bit floating-point multiply-accumulate costs a few picojoules, an SRAM access costs several times more per byte, an off-chip DRAM access costs one to two orders of magnitude more, and writing a byte to NOR flash to persist an updated weight can cost several orders of magnitude more still and wears the cell after limited erase cycles. This is why the intuition "training is 2-3x inference" is dangerously optimistic: measured in energy, a training step that spills activations to DRAM and then commits new weights to flash can cost \(20\times\) or more the energy of the inference it improves.
Key Insight
On-device training is a memory-energy problem disguised as a compute problem. The floating-point math is nearly free; the bill is paid moving activations off-chip and committing weights to non-volatile storage. Every design lever that reduces the energy of a learn step, freezing the backbone so activations need not be stored (Section 62.4), latent replay so you train on small cached embeddings instead of raw windows (Section 62.3), updating one small head instead of the whole network, batching many samples before a single flash commit, works by cutting bytes moved, not by cutting arithmetic. Design for the memory hierarchy first and the FLOP count second.
A joule ledger: budgeting learning against battery life
Turn the physics into a budget. A battery stores a fixed charge; multiply capacity in ampere-hours by nominal voltage to get energy in joules. A CR2032 coin cell holds roughly 220 mAh at 3 V, about 2400 J of usable energy. From that single number you can bound how much learning a device can afford over its target lifetime. If the device must live \(L\) days and inference plus sensing already consumes a baseline power \(P_{\text{base}}\), the energy left for adaptation is
\[ E_{\text{learn budget}} = E_{\text{battery}} - P_{\text{base}}\, L, \]and if each learn step costs \(E_{\text{step}}\), the device can afford at most \(E_{\text{learn budget}} / E_{\text{step}}\) updates across its whole life. Divide by \(L\) and you get a daily update quota. This is the number that should govern the continual-learning policy: not "adapt whenever new data arrives" but "adapt at most \(n\) times per day, and spend those updates where they buy the most accuracy." The listing below computes this ledger for a wearable so the design tradeoff is concrete rather than hand-waved.
# Energy ledger for on-device learning on a coin-cell wearable
mAh, volts = 220.0, 3.0 # CR2032 coin cell
E_battery = mAh/1000 * 3600 * volts # joules = A.s.V = J -> ~2376 J
lifetime_days = 180
# Baseline: always-on inference + sensing draws a steady average current
P_base_uW = 55.0 # microwatts, measured on the board
E_base = P_base_uW * 1e-6 * lifetime_days * 86400
# One TinyOL-style head update: forward + backward + one flash commit
E_compute = 1.4e-3 # joules of MAC + SRAM traffic
E_flash = 2.1e-3 # joules to persist the updated head
E_step = E_compute + E_flash # ~3.5 mJ per learn step
E_learn_budget = E_battery - E_base
max_updates = E_learn_budget / E_step
print(f"battery : {E_battery:8.1f} J")
print(f"baseline draw : {E_base:8.1f} J over {lifetime_days} d")
print(f"budget for learning: {E_learn_budget:8.1f} J")
print(f"affordable updates : {max_updates:8.0f} total "
f"({max_updates/lifetime_days:.0f} per day)")As Listing 62.5.1 shows, the affordable-updates figure is exquisitely sensitive to \(P_{\text{base}}\): if baseline sensing already eats most of the cell, the learning budget collapses toward zero and continual learning must be reserved for rare, high-value moments. It is also sensitive to the flash-commit term, which is why batching updates and persisting weights infrequently (say, once a day rather than once a sample) can multiply the number of gradient steps a device can afford.
The Right Tool
Hand-instrumenting a training loop to attribute energy across compute and memory is hundreds of lines of platform-specific profiling. On mobile and edge SoCs, the CodeCarbon and Zeus libraries wrap a training run and report joules and estimated emissions from the on-die power counters in about five lines, roughly a 20-to-1 reduction over rolling your own RAPL or INA226 reader, and they separate the components for you. On bare-metal microcontrollers there is no such counter, so the honest measurement still comes from a shunt resistor and a power analyser (covered below); the library saves you on anything that runs Linux, and nothing saves you from measuring the real board when picojoules matter.
Deciding when to spend a learn step
A fixed daily quota is only half the policy; the other half is gating, spending each precious update where it earns its charge. The decision rule is a simple expected-value test. Let \(\Delta a\) be the expected accuracy gain from adapting on the current sample or buffer, \(v\) the value of an accuracy point in your application, and \(E_{\text{step}}\) the energy cost. Adapt only when
\[ v\,\Delta a \;>\; \lambda\, E_{\text{step}}, \]where \(\lambda\) is a price on energy that you raise as the battery drains. In practice \(\Delta a\) is unobservable before the fact, so it is proxied by cheap triggers computed during the forward pass you were running anyway: high predictive uncertainty (an entropy or margin threshold, tying to the calibration methods of Chapter 18), a detected distribution shift, or an explicit user correction. When the trigger is quiet, the device stays in pure inference and the learning budget is preserved. This converts "always learning" into "learning on surprise," which is both more accurate and far cheaper. It also composes with duty cycling: defer the actual gradient step to the next scheduled wake or, on a harvested device, to the next moment the buffer is full and the supply is above the training threshold, the charging-aware scheduling idea developed in Chapter 63.
Real-World Application: a hearing aid that learns your rooms without draining by dinner
A premium hearing aid runs a frozen acoustic scene classifier and a small adaptive head that personalises noise suppression to its wearer's environments. Its zinc-air cell holds only a few hundred joules and must last a full day, so continuous on-device fine-tuning is out of the question: measured on the board, one head update costs about 3 mJ, and thousands of them would visibly shorten the day. The firmware instead applies the gate above. It watches the classifier's margin during the forward pass it already runs for every audio frame; only when the wearer enters an acoustically novel room (low margin, sustained) does it cache a few seconds of latent features and, at the next low-load moment, take a single update and commit the head. Over a day that is a few dozen updates, not thousands, costing well under one percent of the cell while still personalising to the restaurant, the car, and the office. The same head, personalised per wearer, is exactly the subject of Section 62.6.
Measuring energy, not estimating it
Every number above is a model, and models of embedded energy are wrong until validated against hardware. The gold standard is direct measurement: put a precise shunt resistor in series with the device supply (or use a source-measure unit or a coulomb counter such as an INA226), log current at high rate, and integrate \(P(t) = V(t)\,I(t)\) over the learn step to get joules. Isolate the step by toggling a GPIO pin high at the start and low at the end so the analyser trace segments cleanly into inference, forward-for-training, backward, and flash-commit phases; you will usually find the flash-commit and any DRAM spill are the tall spikes, confirming the memory-energy story and telling you exactly where to optimise. Datasheet "typical" figures and simulator estimates are for early design only; the moment silicon exists, they are superseded by the shunt. Report energy per learn step and projected days of life the way you would report accuracy, as a first-class metric with its own measurement protocol, so that an "improved" continual-learning method that quietly triples the flash writes cannot pass review as a win.
Research Frontier
The frontier is making the gradient step itself intrinsically cheap. Sparse and structured backpropagation updates only a subset of weights per step, cutting both compute and the bytes committed to flash. Forward-only and perturbation-based learning schemes (the Forward-Forward algorithm and evolutionary or zeroth-order updates) avoid storing activations for a backward pass entirely, attacking the peak-memory term that dominates energy, at some cost in convergence. On the systems side, POET and similar frameworks schedule training on memory-constrained edge devices by paging and recomputing activations to trade a little compute for a large memory saving, and analog in-memory-compute accelerators promise weight updates at a fraction of today's per-byte energy. Across all of them the open question is the honest one for this whole chapter: for a given battery and a given data stream, is any on-device update worth its joules, or is the frugal choice to freeze the model and adapt only the cheap head? The answer is increasingly workload-specific, which is why the measurement discipline above matters more than any single technique.
Exercise
Take a small classifier and a labelled sensor stream on any board you can instrument (a dev kit with a current-sense header, or a Linux SBC with Zeus). First, measure the energy of one inference and one full training step, and report the ratio; comment on whether it matches the 2-3x compute rule or exceeds it because of memory traffic. Second, build the energy ledger of Listing 62.5.1 with your measured \(E_{\text{step}}\) and a battery of your choice, and compute the affordable daily update quota. Third, implement the uncertainty-gated policy: adapt only when prediction margin falls below a threshold, and plot final accuracy versus total joules spent as you sweep the threshold from "never adapt" to "always adapt." Identify the knee of that curve, the point where more energy stops buying meaningful accuracy, and argue for the operating point you would ship.
Self-Check
1. A colleague estimates on-device training energy as "3x the inference FLOPs times the energy per FLOP." Give two reasons this can underestimate the real cost by an order of magnitude, and name the two energy terms that dominate.
2. Using the ledger in Listing 62.5.1, explain why halving the baseline sensing power can increase the affordable number of learn steps far more than halving the per-step compute energy.
3. Write down the expected-value gate for spending a learn step, define each symbol, and explain what practical proxy replaces the unobservable accuracy gain \(\Delta a\) at decision time.
What's Next
In Section 62.6, we spend the energy budget we just learned to protect on the payoff that most justifies it: personalization. The gated, frugal updates of this section are exactly the mechanism by which a shared model becomes your model, adapting to one wearer's gait, one machine's acoustics, one home's routine, without ever sending private sensor data off the device.