"I read the wire before the finger moves. By the time your knuckle bends, I have already spelled the word."
A Neuromotor AI Agent
Prerequisites
This section builds on the transduction and measurement-model view of Chapter 2 and the sampling, bandwidth, and jitter ideas of Chapter 3, since surface EMG lives in a narrow high-frequency band. The windowed features here extend Chapter 8, and the sequence-transduction decoder extends Chapter 14. The deeper electrophysiology of the muscle, including motor-unit decomposition and clinical EMG, is developed in Chapter 32; here we treat EMG strictly as an interaction signal.
The Big Picture
The muscles that move your fingers are not in your fingers. Flexor and extensor digitorum sit in the forearm and pull on tendons that cross the wrist, which means the electrical command to press a key is available at the wrist a few milliseconds before, and sometimes without, any visible motion. A band of dry electrodes worn where a watch sits can read that command as surface electromyography (sEMG) and decode intended keystrokes and gestures directly. This is the premise behind Meta's neuromotor wristband and the public emg2qwerty benchmark: a keyboard you carry on your nerves, usable in the dark, in a pocket, or paired to augmented-reality glasses that have no physical keys at all. This section explains what the signal is, why decoding it looks more like speech recognition than button-press detection, and where the hard problems live.
What surface EMG actually measures
A motor neuron and all the muscle fibers it controls form a motor unit. When the neuron fires, every fiber in that unit depolarizes nearly together, producing a motor-unit action potential (MUAP) that propagates along the fibers. A surface electrode does not see one clean spike; it sees the sum of many MUAPs, each smeared and attenuated by volume conduction through fat and skin, arriving from units at different depths and distances. The recorded voltage is therefore a stochastic superposition: \[ v(t) = \sum_{i} \sum_{k} h_i(t - t_{i,k}) + n(t), \] where unit \(i\) fires at times \(t_{i,k}\), \(h_i\) is that unit's surface signature, and \(n(t)\) is instrumentation and motion noise. Two consequences matter for modeling. First, muscle force is encoded two ways at once, by how fast each unit fires (rate coding) and by how many units are active (recruitment, following the Henneman size principle), so the signal envelope grows with effort but nonlinearly and with hysteresis. Second, the useful information is high-frequency: surface EMG occupies roughly 20 to 450 Hz with amplitudes of tens to a few hundred microvolts, which is why the emg2qwerty bands sample at 2 kHz and why a slow envelope alone throws away most of what distinguishes one finger's intent from another's.
Key Insight
EMG-for-typing is not gesture classification, it is sequence transduction. There is no clean, pre-segmented "press event" to label; keystrokes overlap, roll, and vary in timing, and the number of characters in a window is unknown in advance. The right frame is the one speech and handwriting recognition already solved: map a continuous multichannel stream to an unaligned symbol sequence and let a Connectionist Temporal Classification (CTC) loss discover the alignment. Once you see typing as "acoustic model plus language model over muscle activity," the entire toolbox of Chapter 14 and beam-search decoding transfers directly, and the naive "one classifier per key" approach is revealed as a dead end.
The signal chain and the features that survive it
Between skin and decoder sits a chain that decides whether your data is usable, and each stage is a modeling constraint from Chapter 3. Dry electrodes on a wristband have high, variable skin-contact impedance, so a differential amplifier with strong common-mode rejection is essential to kill the 50/60 Hz mains that swamps the microvolt signal. A band-pass (roughly 20 to 450 Hz) removes motion-artifact drift below and out-of-band noise above, and a notch or adaptive filter attacks mains hum. What reaches the decoder is still non-stationary: electrode shift, sweat, and arm posture change the mixing from units to channels minute to minute. Classical pipelines answer with short sliding windows (about 150 to 250 ms) summarized by time-domain features (mean absolute value, waveform length, zero crossings, slope-sign changes) fed to a linear discriminant classifier, the Hudgins feature set that dominated prosthetics for two decades. Modern systems keep the windowing but let a convolutional network learn the features end to end from the raw multichannel stream, which is what the code below sketches and what the emg2qwerty baselines do at scale.
import numpy as np
def emg_windows(x, fs=2000, win_ms=200, hop_ms=40):
"""Sliding-window time-domain features from multichannel sEMG.
x: (T, C) raw signal in microvolts. Returns (n_windows, C, 3):
per-channel mean-absolute-value, waveform length, zero-crossing rate."""
w, h = int(fs*win_ms/1000), int(fs*hop_ms/1000)
feats = []
for s in range(0, len(x) - w, h):
seg = x[s:s+w] # (w, C)
mav = np.abs(seg).mean(0) # amplitude / effort proxy
wl = np.abs(np.diff(seg, axis=0)).sum(0) # waveform length (complexity)
zc = (np.diff(np.sign(seg), axis=0) != 0).sum(0) / w # zero-crossing rate
feats.append(np.stack([mav, wl, zc], axis=-1)) # (C, 3)
return np.asarray(feats)
rng = np.random.default_rng(0)
fs = 2000
emg = rng.normal(0, 15, (fs, 16)) # 1 s, 16 channels, ~15 uV noise
emg[800:840, 3] += rng.normal(0, 120, (40, 1)) # a burst on channel 3 (a "press")
F = emg_windows(emg, fs)
print(F.shape, "peak-MAV channel:", F[:, :, 0].max(0).argmax())
Right Tool: don't reimplement CTC decoding
The transduction loss and its decoder are the parts you must not hand-roll. torch.nn.CTCLoss trains the alignment in one line, and a beam-search decoder with a character language model (torchaudio.models.decoder.ctc_decoder, the same one used for speech) turns network log-probabilities into text in about 15 lines, replacing several hundred lines of dynamic-programming alignment and beam bookkeeping that are notoriously easy to get wrong on log-sum-exp stability. The official emg2qwerty repository ships exactly this stack on PyTorch Lightning, so a reproducible baseline is a config edit, not a rewrite.
emg2qwerty and the neuromotor wristband
Meta acquired CTRL-labs in 2019 and, in 2024, released emg2qwerty as an open benchmark: roughly 1,135 hours of surface EMG from 108 users touch-typing on a physical QWERTY keyboard, recorded with a wristband on each arm (16 EMG channels per band at 2 kHz), with the keylogger stream giving ground-truth characters and timing. Because the keyboard is real, the labels are precise, yet the decoder must learn to spell from muscle activity alone, and the metric is character error rate (CER) as in speech recognition. The published baseline is a time-depth-separable convolutional encoder trained with CTC and a character language model; a personalized model trained on a given user reaches single-digit CER, while a generic model tested on users it never saw is substantially worse, which is precisely the cross-user gap that Section 27.5 confronts. The productized side of the same technology is Meta's neuromotor wristband, the intended input device for its Orion augmented-reality glasses: rather than full typing it decodes discrete micro-gestures (pinch, thumb swipe, tap) plus continuous cursor control, chosen because they are robust, low-latency, and can be performed with the hand resting at your side.
Research Frontier
The current state of the art on the emg2qwerty benchmark is Meta's own CTC baseline (Sivakumar et al., NeurIPS 2024 Datasets and Benchmarks), and the open problem it exposes is generalization: strong within-user, weak across-user and across-session because electrode placement, anatomy, and skin state shift the source-to-channel mixing. Active directions are self-supervised pretraining on unlabeled EMG (the contrastive methods of Chapter 17), rotation-and-shift augmentation to synthesize band displacement, and rapid few-shot calibration. Meta's public 2025 result of near-typing-speed generic decoding with only brief personalization is the headline claim the field is now trying to reproduce and extend, and it is the direct subject of the next section.
In Practice: a clinical stroke-rehab keyboard with no keys
A rehabilitation clinic piloted the wristband for patients recovering partial hand function after stroke. Conventional assistive typing needed visible, well-formed finger movements the patients could not yet produce reliably, so progress stalled and motivation sagged. Surface EMG changed the target: because the wrist band reads the descending motor command, a patient who could generate the intent to move an index finger, even a weak, barely-visible twitch, produced a decodable burst on the flexor channels. The team trained a small per-patient CTC decoder and mapped clean, repeatable bursts to a reduced key set, giving a communication channel that scaled with recovery rather than demanding full dexterity up front. Two lessons generalized. First, per-user calibration was not a nuisance but the whole point: each patient's residual signal was idiosyncratic, and a generic model would have erased exactly the individual structure that mattered. Second, leakage discipline was non-negotiable; an early accuracy number looked wonderful until the team realized adjacent windows from one session had leaked across the train and test split, inflating the score, a textbook failure the subject-wise protocols of Chapter 5 exist to prevent.
Why EMG beats the alternatives, and where it does not
Against the IMU gestures of Section 27.2 and the camera- and glove-based pose sensing of Section 27.3, surface EMG offers three things none of them can: it works with the hand hidden or still, it reads intended force and micro-movements below the threshold of visible motion, and it is private in a way a camera is not. The costs are equally real. The signal is non-stationary and drifts within a session; dry-electrode contact is fragile to sweat and band rotation; and cross-user transfer is genuinely hard because every forearm mixes its motor units onto the electrodes differently. This is why the deployed wristband leans on a small, robust gesture vocabulary rather than full free-text typing, and why the research race is about closing the calibration gap. EMG is at its best not as a lone modality but fused with an IMU for coarse arm pose and a camera when available, the multimodal-interaction theme of Section 27.6, and its full electrophysiological treatment, including high-density arrays and motor-unit decomposition, waits in Chapter 32.
Exercise
You have emg2qwerty-style data: 16-channel sEMG at 2 kHz from both wrists, with character labels and timestamps. (1) Explain why framing this as per-key classification over fixed windows fails, and why a CTC objective does not need aligned per-character boundaries. (2) The classical Hudgins pipeline uses about 200 ms windows; what goes wrong if you shrink the window to 20 ms, and what goes wrong at 600 ms, in terms of the rate-coding and recruitment content of the signal? (3) Design a train/validation/test split for the generic (cross-user) setting, and state exactly which grouping variable you must split on so that a within-session correlation cannot leak (see Chapter 5).
Self-Check
1. The muscles that flex your fingers are in the forearm, not the hand. Why does that single anatomical fact make a wrist-worn EMG band a viable finger-intent sensor?
2. Surface EMG encodes muscle force through two mechanisms at once. Name both, and say why they make the amplitude-to-force relationship nonlinear.
3. Why is character error rate, borrowed from speech recognition, a more honest metric for EMG typing than per-key accuracy?
What's Next
In Section 27.5, we attack the gap that emg2qwerty makes unavoidable: a decoder that is excellent for the user it was trained on collapses on a stranger's forearm. We turn to cross-user, calibration-free neuromotor decoding, where self-supervised pretraining, domain-shift-aware augmentation, and few-shot adaptation try to make a wristband work the moment you put it on, with no per-user training session at all.