IMU gestures and low-latency inference

"The wearer flicked a wrist and expected the world to have already answered. My whole job was to decide what they meant before they finished meaning it."

A Low-Latency AI Agent

What an IMU gesture is, and the always-on spotting problem

An IMU gesture is a short, intentional, discrete movement drawn from a small vocabulary: wrist flick, double-tap, wake-on-raise, shake-to-undo, an air-drawn digit, a controller swipe. Unlike an activity, it has a defined onset and offset and it is rare. That rarity is the central design pressure. A wrist-worn recognizer runs continuously, sampling the accelerometer and gyroscope at 50 to 200 Hz, and for the overwhelming majority of that time the correct answer is "no gesture." The task is therefore not classification but gesture spotting: continuously detecting whether a gesture is occurring, where it starts and ends, and only then which one it is. This is open-set detection against an unbounded NULL class of everyday motion, the same asymmetry seen in Chapter 26 but far more extreme because the positive events are seconds apart.

The metric that governs a spotter is not accuracy but the false-activation rate, usually quoted as false positives per hour of ordinary wear. A raise-to-wake feature that fires spuriously twice an hour is unshippable, because each false wake drains battery and erodes trust, and the user cannot tell the system "no." So the operating point sits deep on the high-precision side of the curve, and the true enemy is the confuser: reaching for a coffee cup looks a lot like a deliberate raise. Handling this well means training with an enormous, diverse negative set of real daily motion, not just clean gesture recordings, and it means calibrating a confidence threshold rather than trusting a raw softmax, a theme we return to through the conformal methods of Chapter 18.

The latency budget: where the milliseconds go

"Low latency" is meaningless until you decompose it. The user-perceived, motion-to-response latency is a sum of stages, and a model that is fast in isolation can still feel sluggish because it is starved of context by the stages around it:

Here \(T_{\text{sample}}\) is the wait for the next IMU sample and its transport off the sensor (often the smallest term, but non-zero at 50 Hz where samples are 20 ms apart). \(T_{\text{window}}\) is the analysis window the model needs to see the gesture; a 250 ms window means the decision cannot, in principle, be made until at least 250 ms of motion has accrued. \(T_{\text{infer}}\) is the forward pass. \(T_{\text{debounce}}\) is the deliberate delay added to suppress repeats and confirm the event. \(T_{\text{actuate}}\) is the downstream response, up to the display or haptic. The uncomfortable truth is that \(T_{\text{window}}\) usually dominates, so the fight for latency is mostly a fight to shrink or partially overlap the window, not to shave microseconds off the network.

How fast is fast enough? Perceptual research gives rough anchors: below about 100 ms an interface feels instantaneous, and around 1 ms of controller-to-render latency is the target for AR and VR to avoid motion sickness. For discrete wearable gestures a common engineering budget is a sub-150 ms end-to-end response, which after subtracting actuation and debounce leaves the model and its window fighting over perhaps 80 ms. That constraint, not the dataset, is what forces the architecture. It also forces causal filtering: the zero-phase filters of Chapter 6 that look into the future are forbidden, so denoising must be strictly causal and its group delay counts against the budget.

Streaming, causal models, and early commitment

Two implementation styles dominate. The sliding-window approach re-runs a fixed-length classifier on the most recent \(W\) samples every stride, which is simple and maps onto the temporal convolutional and recurrent models of Chapter 14, but it recomputes overlapping work and its latency is floored by \(W\). The streaming-state approach keeps a recurrent hidden state or a causal convolution buffer and advances it one sample at a time, so per-sample cost is constant and the model can emit a decision at any instant. Streaming state is why linear-time recurrent and state-space models are attractive on wearables; the broader streaming-inference machinery, including how to warm and checkpoint that state, is the subject of Chapter 60.

On top of either style sits the commitment logic. A robust spotter does not fire the moment one frame's probability spikes; it waits until the top class has held above a threshold \(\tau\) for a short confirmation, then fires once and enters a refractory period during which it will not fire again, exactly like a debounced button. This suppresses the double-taps and mid-gesture flickers that otherwise turn one flick into three events. The expected latency of such a policy is the time for accumulated evidence to cross \(\tau\), which you can lower by lowering \(\tau\) at the cost of more false activations. The code below implements this streaming spotter with a stub scorer so the control logic is visible; swapping in a real quantized network changes only the score call.

import numpy as np
from collections import deque

class StreamingGestureSpotter:
    """Causal, one-sample-at-a-time IMU gesture spotter with early
    commitment and a refractory period. Runs in O(1) work per sample."""
    def __init__(self, win=25, tau=0.85, confirm=3, refractory=20, n_cls=4):
        self.buf = deque(maxlen=win)     # causal ring buffer of recent samples
        self.tau, self.confirm = tau, confirm
        self.refractory = refractory
        self.above = 0                   # consecutive frames top class > tau
        self.cooldown = 0                # samples left in refractory
        self.last_top = -1
        self.n_cls = n_cls

    def score(self, window):
        """Stub for a real quantized net: returns class probabilities.
        Here: a cheap energy/axis heuristic just to exercise the logic."""
        w = np.asarray(window)
        feat = np.array([w[:, 0].std(), w[:, 1].std(),
                         w[:, 2].std(), 1.0])          # 4th = NULL bias
        logits = feat - feat.mean()
        e = np.exp(logits - logits.max())
        return e / e.sum()

    def step(self, sample):
        """Feed one IMU sample (ax, ay, az); return fired class id or None."""
        self.buf.append(sample)
        if self.cooldown > 0:
            self.cooldown -= 1
            return None
        if len(self.buf) < self.buf.maxlen:
            return None                  # not enough history yet
        p = self.score(self.buf)
        top = int(p.argmax())
        if top != self.n_cls - 1 and p[top] >= self.tau and top == self.last_top:
            self.above += 1
        else:
            self.above = 0
        self.last_top = top
        if self.above >= self.confirm:   # early commit once evidence holds
            self.above, self.cooldown = 0, self.refractory
            return top
        return None

spotter = StreamingGestureSpotter()
rng = np.random.default_rng(0)
stream = rng.normal(0, 0.1, size=(200, 3))
stream[80:100, 0] += 1.5                 # inject a class-0 gesture burst
fires = [(t, spotter.step(s)) for t, s in enumerate(stream)]
print([f for f in fires if f[1] is not None])

Making it fit and stay honest

The model that satisfies this budget is almost always tiny: a small 1D temporal convolutional network or a light recurrent model, quantized to 8-bit integers and running on the wearable's microcontroller or low-power DSP, using the compression and kernel-fusion techniques of Chapter 59. A frequent and effective pattern is the two-stage cascade the earbud team used: a nearly free always-on trigger (an energy or jerk threshold) gates a heavier classifier that runs only on candidate segments, cutting average power by an order of magnitude while preserving worst-case latency on real gestures.

Evaluation must respect the same asymmetry the metric does. Report latency as a distribution, not a mean: the median tells you how it usually feels, but the 95th percentile tells you how often it feels broken, and a long tail from occasional recomputation is what users actually complain about. Report false activations over long, unscripted, real-world wear, not over a balanced test set, because a balanced set hides the base-rate problem that dominates always-on use. And evaluate on held-out users and sessions: gesture styles are idiosyncratic, and a spotter that memorizes one person's exact flick trajectory will fail on the next wrist, the leakage-safe discipline that runs through this whole book and is formalized in Chapter 5. The cross-user generalization problem is severe enough that Section 27.5 devotes itself to calibration-free decoding.

What's Next

In Section 27.3, we move from recognizing a handful of discrete gestures to continuously reconstructing hand and body pose from wearable sensors. The step up is large: instead of committing to one of a few labels, the system must regress a full articulated configuration in real time, which raises the same latency questions we met here but against a far richer, continuous output space.