Non-invasive language/speech decoding

"Give me a scalp electrode and enough teacher forcing, and I will reconstruct Hamlet from a coin flip."

A Skeptical AI Agent

This section assumes the montage, band, and artifact material from Section 31.1 and Section 31.2, the foundation-model encoders from Section 31.4, and the contrastive-learning machinery from Chapter 17. If any of those are unfamiliar, read them first; the decoders below are thin heads on top of exactly that stack.

Three problems that all get called "mind reading"

Vague press coverage collapses three very different tasks. Separating them is the single most useful thing you can do before reading any paper. Speech perception decoding asks: while a subject listens to audio, can we identify which speech segment they heard? Imagined or inner-speech decoding asks: while a subject silently rehearses words, can we recover them? Semantic reconstruction asks: can we produce text whose meaning tracks a story the subject is hearing or watching, without word-for-word fidelity? These differ in difficulty by orders of magnitude. Perception decoding has a strong, time-locked stimulus and is the easiest. Inner speech has no external anchor, weak and idiosyncratic signals, and is the hardest. Semantic reconstruction sits between, and crucially it is evaluated on meaning overlap, not exact strings, which changes what counts as success.

Contrastive brain-to-speech alignment

The workhorse architecture borrows directly from CLIP-style multimodal learning (see Chapter 21). A brain encoder maps a windowed multichannel recording \(x \in \mathbb{R}^{C \times T}\) to an embedding \(z_b\); a frozen self-supervised speech model such as wav2vec 2.0 maps the concurrent audio to \(z_a\). Both are L2-normalized and trained so that matching brain-audio pairs align while mismatched pairs repel, using the symmetric InfoNCE objective

where \(s\) is cosine similarity and \(\tau\) a learned temperature. The brain encoder is a subject-conditioned convolutional or transformer stack: a per-subject spatial layer absorbs individual electrode geometry and skull anatomy, then a shared backbone does the temporal work. This subject-embedding trick is what lets one model pool many people despite each brain being wired slightly differently, and it connects directly to the adaptation methods in Section 31.7. At inference you embed the brain segment and retrieve the nearest audio candidate. No text is generated, and there is nothing to leak.

import torch, torch.nn.functional as F

def clip_brain_speech_loss(z_brain, z_audio, temperature=0.07):
    # z_brain, z_audio: (N, D) segment embeddings from the same time windows
    zb = F.normalize(z_brain, dim=-1)
    za = F.normalize(z_audio, dim=-1)
    logits = (zb @ za.t()) / temperature      # (N, N) similarity matrix
    targets = torch.arange(z_brain.size(0), device=z_brain.device)
    # symmetric: brain->audio and audio->brain both classify the diagonal
    return 0.5 * (F.cross_entropy(logits, targets) +
                  F.cross_entropy(logits.t(), targets))

# retrieval eval: does the true clip rank in the top-k for each brain window?
@torch.no_grad()
def topk_retrieval(z_brain, z_audio, k=10):
    sims = F.normalize(z_brain, -1) @ F.normalize(z_audio, -1).t()
    ranks = sims.argsort(dim=-1, descending=True)[:, :k]
    hit = (ranks == torch.arange(len(z_brain))[:, None]).any(-1)
    return hit.float().mean().item()

Why EEG-to-text papers overclaimed

A wave of 2022 to 2023 papers reported fluent open-vocabulary sentence generation from EEG (the DeWave line and its relatives), with BLEU and ROUGE scores that looked transformative. In 2024 the results were shown to be an evaluation artifact. The sequence-to-sequence decoders were trained and, critically, tested with teacher forcing: at each step the model received the previous ground-truth token as input. A language decoder with teacher forcing produces fluent, on-topic text from noise, because it is really doing next-token prediction conditioned on the reference. Replace the EEG input with zeros or random values and the scores barely move. The lesson is a leakage lesson, the same family of failure catalogued in Chapter 5: if a control condition with the signal removed still "works," the signal was never the source of performance.

Modalities, and the semantic-reconstruction frontier

Modality choice sets the ceiling. EEG is portable and cheap but spatially smeared and artifact prone; it supports small closed vocabularies and coarse category decoding. MEG has far cleaner spatiotemporal structure and drives the best perception-retrieval numbers, but the machine is room sized and non-portable. fNIRS measures slow hemodynamics, useful for detecting speech intent or answering binary questions, not for word-rate decoding. fMRI is the outlier: Tang and colleagues (2023, Nature Neuroscience) reconstructed the gist of heard and imagined stories by pairing a GPT-class language model with a per-subject encoding model and beam search over candidate continuations, scored by predicted BOLD response. It reconstructs meaning, not words, requires roughly 16 hours of per-subject scanner training, and, notably, failed when the subject actively performed a distractor task, an inadvertent demonstration of mental privacy.

What's Next

In Section 31.7, we turn from decoding accuracy to the surrounding obligations: subject adaptation so a model built on many brains works on a new one with minutes of calibration, safety and mental-privacy safeguards for signals that (as the distractor result showed) can be withheld by the user, and interpretability methods that tell you which cortical sources and frequency bands a decoder actually relies on before anyone trusts it in the clinic.