Human factors, usability, and latency budgets

"A user will forgive me for guessing wrong long before they forgive me for guessing late. Make me fast, then make me right, and never confuse the order."

An Impatient AI Agent

The latency budget: accounting for every millisecond

The single most important number in an interactive system is end-to-end latency, the time from the user's motor act to the perceivable response. It is a sum of stages, and each stage is owned by a different part of the stack:

Here \(t_{\text{sense}}\) is the sampling and analog settling time, \(t_{\text{buffer}}\) is the window your model needs before it can even decide (a gesture recognizer that consumes 150 ms of IMU history has already spent 150 ms), \(t_{\text{infer}}\) is model compute, \(t_{\text{comm}}\) is any wireless or bus hop (Bluetooth Low Energy adds a connection-interval-quantized 7.5 to 30 ms all by itself), and \(t_{\text{render}}\) plus \(t_{\text{display}}\) are the UI and panel refresh. Why does the exact number matter so much? Because human perception has sharp thresholds. Under roughly 20 ms, a dragged object feels glued to the finger; by 100 ms the system still feels "responsive" for discrete actions but direct manipulation feels rubbery; past 100 ms of head-to-display latency in a headset, users get motion sick. Nielsen's classic tiers (0.1 s feels instant, 1 s keeps flow, 10 s loses attention) are the coarse version; interaction designers work an order of magnitude finer.

The subtle, expensive mistake is to budget for the mean. Users do not perceive averages; they perceive the worst hitches. A pipeline whose median latency is a comfortable 45 ms but whose 99th percentile is 180 ms will feel intermittently broken, and the culprit is usually variance, not the mean: a garbage-collection pause, a Bluetooth retransmit, a thermally throttled core. You must budget the tail. The code below models each stage as a distribution and reports the percentiles that users actually feel.

import numpy as np

# Each stage: (name, mean_ms, jitter_ms). Buffer is deterministic; BLE and GC are heavy-tailed.
stages = [
    ("sense",   2.0,  0.3),
    ("buffer", 40.0,  0.0),   # fixed model context window
    ("infer",   6.0,  1.0),   # on-device quantized net
    ("comm",   12.0,  9.0),   # BLE connection interval, occasional retransmit
    ("render",  4.0,  1.5),
    ("display", 8.3,  4.8),   # ~120 Hz panel, phase-random wrt event
]

rng = np.random.default_rng(0)
N = 200_000
total = np.zeros(N)
for name, mu, jit in stages:
    # Gamma keeps latency positive and gives a realistic right tail.
    shape = (mu / jit) ** 2 if jit > 0 else 0.0
    s = rng.gamma(shape, jit**2 / mu, N) if jit > 0 else np.full(N, mu)
    total += s
for p in (50, 95, 99, 99.9):
    print(f"p{p:<4} end-to-end latency: {np.percentile(total, p):6.1f} ms")
print(f"fraction over 100 ms budget: {100*np.mean(total > 100):.2f}%")

Throughput and the asymmetric cost of errors

Speed and accuracy combine into throughput, the real currency of an input device. For pointing and selection tasks the governing law is Fitts's law: the movement time to acquire a target of width \(W\) at distance \(D\) is

where the logarithmic term is the index of difficulty \(ID\) in bits, and \(a, b\) are device-and-user constants fit by regression. The ISO 9241-411 standard formalizes this into a repeatable multidirectional-tapping test, and it is how you compare a wristband cursor against a trackpad against eye-gaze on equal footing: measured in bits per second, a good mouse reaches roughly 4 to 5 bit/s, mid-air hand gesture typically 2 to 3, and early neuromotor cursors are climbing into that range. Fitts's law is also a design tool: it tells you that making targets bigger (\(W \uparrow\)) or closer (\(D \downarrow\)) helps more, per bit, than shaving raw latency, which is why good gesture UIs use large forgiving targets rather than demanding pixel precision from a noisy sensor.

Then there is the error the leaderboard hides: the false activation. In a gesture system, false positives and false negatives are not symmetric. A missed swipe costs one repeat; a spurious "delete" or an accidental unlock costs trust, and possibly data or money. This is the same operating-point choice as calibrated rejection in Chapter 18: you set the decision threshold not at the equal-error point but where the expected cost is minimized, given that a false activation may be ten or a hundred times worse than a miss. Always-on gesture sensing makes this acute because the "null" class (the user doing anything other than gesturing) is enormous and adversarial, so a 0.1% false-positive rate per second is dozens of phantom commands a day.

Fatigue, the Midas touch, and designing the interaction

Two human-factors failure modes recur across every modality in this chapter, and both are invisible in an offline metric. The first is fatigue, epitomized by "gorilla arm": sustained mid-air interaction against gravity is exhausting because the shoulder is a poor holding actuator. This is why the best pose and neuromotor interfaces, including the wristband electromyography of Section 27.4, deliberately favor small, low-effort, hand-and-finger micro-gestures resting in the lap over expansive arm waving. Comfort is a design constraint you can measure (with instruments like the Borg exertion scale or the Consumed Endurance metric), and it should appear in your evaluation protocol next to accuracy.

The second is the Midas-touch problem: when the sensor is always on, how does it know when you mean to command it versus merely moving? Every practical always-on interface solves this with an explicit segmentation strategy rather than a better classifier: an arming gesture or wake motion, a dead-zone the signal must exceed, a required dwell or a deliberate onset shape, or a physical clutch (a button, a pinch-to-engage). These are cheap, and they buy more perceived reliability than another point of model accuracy, because they convert a hard open-set problem into an easier gated one. Good interaction design also closes the loop: immediate, unambiguous feedback (a haptic tick, a visual highlight, an audio cue) at the moment of recognition lets the user build an accurate mental model of what the system heard, which is what actually makes an imperfect recognizer feel trustworthy.

Measuring usability and designing for everyone

Beyond speed and errors sit the qualities that decide whether a first-time user succeeds and a long-term user stays: learnability (how fast a novice reaches competence), memorability (whether they retain the gestures after a week away), guessability (whether the natural first attempt is the right one), and overall satisfaction. These are measured with real people, using instruments you should treat as seriously as your test set: the System Usability Scale (SUS) for a comparable 0 to 100 score, task-completion and error rates from a supervised study, and structured elicitation studies to choose gestures users actually propose rather than ones an engineer invented. Crucially, evaluate with a held-out population of users, not the three colleagues who helped train the model, for exactly the leakage reasons that the cross-user work of Section 27.5 and the activity-recognition splits of Chapter 26 insist on.

Finally, human factors means all humans. Gesture and neuromotor interfaces have an extraordinary accessibility upside, offering input paths for people who cannot use a keyboard, mouse, or touchscreen, but only if you design for variation rather than a median 25-year-old engineer. Tremor, limited range of motion, a prosthesis, one-handed use, and differing skin, limb, and physiology all shift the signal distribution. The design responses are concrete: adjustable thresholds and dwell times, personalization and quick recalibration, redundant input paths, and never gating a critical action behind a single gesture the user might be unable to perform. Accessibility is not a compliance afterthought; it is the same distribution-shift robustness problem the whole book keeps returning to, pointed at the most important out-of-distribution population of all.

What's Next

In Chapter 28, we cross from interaction into physiology. The same electrodes and inertial units that decoded intent now measure the body's own signals, ECG, PPG, EEG, EMG, respiration, and electrodermal activity, where the "latency budget" gives way to a "reliability and validation budget" and a wrong answer is no longer a missed swipe but a missed diagnosis, raising the stakes on every idea from this part.