Invasive BCI: speech and handwriting decoding (Neuralink, Synchron, cortical arrays)

"A thousand electrodes will happily read your motor cortex. Getting the comma in the right place still takes a language model."

An Implanted AI Agent

This section builds on the montage-and-band vocabulary of Section 31.1 and the motor-imagery decoding of Section 31.3, but pushes past the scalp. It leans on the sequence-model toolkit of Chapter 14 (RNNs and temporal convolutions) and the language-model reasoning of Chapter 21. No neurosurgery background is assumed; channel counts, firing rates, and typing speeds here are engineering anchors for sizing decoders, not clinical claims. The regulatory and safety weight of putting hardware in a human brain is treated in Chapter 34.

Why go inside the skull: the fidelity ladder

Invasiveness buys signal. Rank the recording modalities and the tradeoff is stark. Scalp EEG sees roughly 1 to 10 microvolts through bone and gives you frequency-band power over centimeters of cortex. Electrocorticography (ECoG), a subdural grid of electrodes resting on the cortical surface, sees millivolt-scale local field potentials and, critically, clean high-gamma power (roughly 70 to 170 Hz) whose amplitude tracks local population firing on a 10-millisecond timescale. Penetrating microelectrode arrays go one level deeper, resolving the action potentials of individual neurons: the classic Utah array (Blackrock Neurotech) is a 4 by 4 millimeter bed of 96 silicon needles, each tip recording spikes from a handful of nearby cells.

The reason this matters for language is bandwidth. Intended movement (whether of a hand, a pen, or the vocal tract) is encoded in the fine-grained, fast-changing firing of motor cortex. Scalp EEG throws almost all of that away, which is why non-invasive BCIs top out at a handful of commands (Section 31.3). An intracortical array preserves enough of it to distinguish 26 imagined letters or dozens of phonemes. The engineering cost is equally stark: surgery, biocompatibility, and slow signal decay as scar tissue encapsulates the electrodes. This is the invasiveness-versus-fidelity dial, and every device below sits at a different point on it.

The hardware zoo: arrays, threads, and a stent

Three sensor architectures dominate the current clinical landscape, and they differ mainly in how they trade channel count against surgical risk.

Cortical microelectrode arrays (BrainGate). The academic workhorse is one or two Utah arrays placed in motor cortex, feeding the long-running BrainGate consortium studies. Roughly 100 to 200 channels of threshold-crossing spikes and spike-band power. High fidelity, decades of results, but rigid needles and a percutaneous connector.

Neuralink N1. Neuralink's implant replaces rigid needles with about 64 flexible polymer threads carrying roughly 1024 electrode contacts, inserted by a surgical robot to dodge blood vessels, and reads out wirelessly with no skull-penetrating connector. Its first human participants (the PRIME study, from 2024) used motor-cortex signals for cursor and click control; the same high channel count is the substrate the company targets for text output.

Synchron Stentrode. Synchron sits at the low-invasiveness end. Its Stentrode is an expandable stent studded with about 16 electrodes, threaded through the jugular vein into the superior sagittal sinus, a vein that runs over motor cortex. No open-brain surgery: it is placed like a cardiac stent. The tradeoff is fewer, vein-wall-distance electrodes and lower spatial resolution, traded for a far lower surgical bar (the COMMAND early-feasibility study).

From imagined handwriting to text

The result that made intracortical text real was handwriting, not typing. Willett and colleagues (Nature, 2021) had a paralyzed BrainGate participant imagine handwriting individual letters. Attempted pen strokes produce richer, more separable motor-cortex trajectories than imagined point-and-click typing, so the neural patterns for "a" versus "b" are easier to tell apart. A recurrent network decoded the array signal into a stream of character probabilities; the raw decoder hit about 94 percent character accuracy at roughly 90 characters per minute, and a language-model autocorrect pass lifted it above 99 percent.

The decoding recipe is worth naming because it recurs across every system above. First, extract a per-channel feature at a modest frame rate (20-millisecond bins of threshold crossings and spike-band power). Second, run a recurrent or temporal-convolutional network (Chapter 14) that emits, at each frame, a distribution over output symbols plus a "blank." Third, collapse that frame-wise output into a symbol string with connectionist temporal classification (CTC), which handles the fact that you never know exactly which frame a letter starts on. Fourth, rescore candidate strings with a language model so that neurally ambiguous decodes resolve toward real words. Listing 31.5.1 shows the feature and CTC-collapse stages on synthetic data so the shapes are concrete.

import numpy as np
rng = np.random.default_rng(0)

# === Stage 1: neural feature extraction (the invasive-BCI primitive) ===
# Simulate 20 ms of raw broadband from 96 channels at 30 kHz.
fs, n_ch, dur = 30_000, 96, 0.20
raw = rng.standard_normal((n_ch, int(fs * dur)))
# Threshold-crossing rate = spikes below -4.5 x RMS, the canonical intracortical feature.
thr = -4.5 * raw.std(axis=1, keepdims=True)
crossings = np.sum(np.diff((raw < thr).astype(int), axis=1) == 1, axis=1)
sbp = np.mean(raw ** 2, axis=1)  # spike-band power, the second standard feature
print("threshold-crossing counts (first 5 ch):", crossings[:5])

# === Stage 4: CTC greedy collapse of a decoder's per-frame symbol argmax ===
# Vocabulary: blank(0), then a,b,c. Suppose the RNN emitted this frame path:
vocab = ["_", "a", "b", "c"]          # "_" is the CTC blank
path  = [1, 1, 0, 1, 2, 2, 0, 2, 3]   # per-frame argmax over 9 frames
def ctc_collapse(seq):
    out, prev = [], None
    for s in seq:
        if s != 0 and s != prev:      # drop blanks and merge repeats
            out.append(vocab[s])
        prev = s
    return "".join(out)
print("decoded string:", ctc_collapse(path))   # -> "abac"

As Listing 31.5.1 makes clear, the invasive part is entirely in Stage 1: nothing in the CTC machinery cares whether the frames came from a Utah array or a Stentrode. That modularity is why the same decoder architecture, retuned, drives handwriting, cursor control, and speech.

import torch
from torchaudio.models.decoder import ctc_decoder

loss = torch.nn.CTCLoss(blank=0)              # replaces ~120 lines of forward-backward
decoder = ctc_decoder(lexicon="words.txt",    # replaces ~200 lines of beam + LM fusion
                      tokens=["_", "a", "b", "c"],
                      lm="brain_text.bin", beam_size=50)
# hypotheses = decoder(emissions)   # emissions: (batch, time, n_symbols) log-probs

One methodological warning carries over from the rest of the book. These systems are trained per-participant on limited, expensive sessions, and neural signals drift day to day as electrodes settle and scar. Reporting accuracy on the same block you tuned on is leakage; honest evaluation holds out whole sessions and reports across days, which is exactly the calibration-and-drift discipline of Chapter 18. The 99-percent handwriting number is a held-out-sentence number with the language model in the loop, not a raw single-frame accuracy.

What's Next

In Section 31.6, we come back out through the skull: non-invasive language and speech decoding, where fMRI and MEG-based systems reconstruct the gist of perceived and imagined language without surgery, at a fraction of the fidelity, and where the privacy and consent questions raised by reading language from the brain get sharper precisely because the sensor no longer requires an operating room.