Sensor-to-text: captioning and describing signals

"I was asked to describe the signal. I wrote 'the patient is definitely walking briskly on a treadmill.' The strap had fallen off two minutes earlier. Nobody checked the timestamp but me, and I lied."

A Loquacious AI Agent

Captioning is the sensor analogue of image captioning, but with a crueler evaluation problem: a reader can glance at a photo and judge whether "a dog on a beach" is right, yet almost nobody can look at a 200 Hz triaxial accelerometer trace and verify "descending stairs." The ground truth lives in metadata, video, or a clinician's note, never in the signal a human can eyeball. That asymmetry shapes every design choice below, and it is why grounding and verification earn their own section (Section 21.6).

What "describing a signal" actually means

What we are building is a conditional language model \(p_\theta(\mathbf{y} \mid \mathbf{x})\) that emits a token sequence \(\mathbf{y} = (y_1, \dots, y_T)\) given a raw sensor window \(\mathbf{x} = \{x_c(t)\}\). Why this is not just classification with extra steps: a label is one draw from a fixed, closed vocabulary, while a caption composes an open-ended description across several axes at once. A good sensor caption typically layers three things: (1) a semantic event ("brushing teeth"), (2) quantitative signal properties ("dominant energy near 2 Hz, amplitude growing over the window"), and (3) context and caveats ("brief signal dropout at second 12; motion artifact likely"). Layer (2) is what makes sensor captioning different from image captioning: the caption is expected to be numerically literate about the physics, not merely name the activity.

How the levels differ in ambition matters for how you train. The weakest useful captioner is templated: a classifier plus a slot-filling sentence ("Detected {activity} for {duration}s at {confidence}"). It never hallucinates because it never composes, but it also cannot say anything the template author did not anticipate. The strongest is a free-form generative captioner that can describe events no label set enumerated, at the cost of being able to invent them. Most production systems live in between, and knowing where you sit on that spectrum is the single most important decision in the section.

Architectures: prefix mapping versus adapter fusion

Two families dominate. The first, and by far the most practical, is the prefix (or soft-prompt) mapping approach: freeze a pretrained language model, freeze (or lightly tune) the sensor encoder from Section 21.1, and train only a small projection network \(g_\phi\) that turns the encoder output into a short sequence of "virtual tokens" prepended to the text prompt. The language model then decodes as if those vectors were words. This is the sensor cousin of how prefix-tuned image captioners and IMU2CLIP-style encoders bolt a perception module onto a frozen decoder. Why it wins in practice: it touches only a few million trainable parameters, so it fine-tunes on a modest sensor-caption corpus without catastrophically forgetting the language model's fluency, and it keeps the expensive decoder reusable across sensor modalities.

The second family, cross-attention adapter fusion, inserts trainable cross-attention layers inside the decoder so every generated token can attend back to the full sequence of sensor patch embeddings, not just a fixed-length prefix. This preserves fine temporal detail (useful when a caption must localize "a stumble near second 38"), at the cost of more trainable parameters and a decoder you can no longer swap freely. Choose prefix mapping when you want a portable, cheap captioner over window-level summaries; choose adapter fusion when the caption must reference specific moments within a long window.

import torch, torch.nn as nn

class PrefixSensorCaptioner(nn.Module):
    """Map a frozen sensor embedding into K virtual tokens for a frozen LLM."""
    def __init__(self, sensor_encoder, llm, d_sensor=256, k_tokens=8):
        super().__init__()
        self.encoder = sensor_encoder.eval().requires_grad_(False)  # from 21.1
        self.llm = llm.eval().requires_grad_(False)                 # frozen decoder
        d_llm = llm.config.hidden_size
        # The ONLY trainable part: a projection into k virtual tokens.
        self.project = nn.Sequential(
            nn.Linear(d_sensor, d_llm), nn.GELU(),
            nn.Linear(d_llm, k_tokens * d_llm))
        self.k, self.d_llm = k_tokens, d_llm

    def prefix(self, window):                 # window: (B, C, T) raw sensor
        with torch.no_grad():
            z = self.encoder(window)          # (B, d_sensor)
        return self.project(z).view(z.size(0), self.k, self.d_llm)  # (B, K, d_llm)

    def forward(self, window, prompt_embeds):
        soft = self.prefix(window)            # sensor "words"
        inputs = torch.cat([soft, prompt_embeds], dim=1)
        return self.llm(inputs_embeds=inputs).logits   # next-token logits

The snippet above makes the parameter economy concrete: everything is frozen except project, so the gradient only ever flows through a two-layer MLP. That is why this pattern trains on a laptop-scale corpus while a full multimodal pretraining run does not.

Where captions come from: the data bottleneck

The hard part is never the architecture; it is the supervision. Sensor-caption pairs do not exist in the wild the way image-alt-text pairs do, so teams manufacture them. Three recipes recur. First, label-to-caption templating: take an existing labeled activity dataset and expand each label plus computed features into a sentence, optionally rewritten by an LLM for variety. Second, feature narration: compute an interpretable feature panel (dominant frequency, RMS trend, zero-crossing rate, peak intervals from Chapter 8) and prompt a strong LLM to narrate those numbers into prose, which grounds the caption in real measurements rather than free invention. Third, expert annotation, expensive but essential for clinical work. Recent sensor-language efforts such as the SensorCaps corpus (the captioning dataset introduced alongside the LLaSA work) combine templating and LLM narration precisely because pure human annotation does not scale to the token counts an LLM decoder needs.

Making captions numerate and calibrated

A caption that says "high-frequency vibration" is weaker than one that says "energy concentrated at 118 Hz, near the second bearing harmonic," and the difference is whether the caption carries real measurements. How to enforce this: feed the decoder not only the learned sensor embedding but a small, explicit tool panel of computed scalars (spectral peaks, trend slopes, event counts) as text, so the model narrates numbers it did not have to hallucinate. This is a preview of the retrieval-and-tool pattern in Section 21.5, and it dramatically cuts numeric fabrication. The second discipline is calibrated hedging: a captioner should say "likely" when the encoder embedding sits far from any training cluster, and "clearly" when it sits in a dense one. Coupling caption confidence words to a conformal or ensemble uncertainty score (Chapter 18) turns the caption's hedging from a stylistic tic into a trustworthy signal.

What's Next

In Section 21.3, we move from monologue to dialogue: instead of the model describing a signal unprompted, a human (or an agent) asks a specific question about it. We will meet the sensor question-answering benchmarks (SensorQA, SensorBench, OpenSQA), see how a captioner becomes the perception front end of a QA system, and learn why answering "did the patient fall?" is a sharper and more honest test of grounding than generating a paragraph nobody can fully check.