Aligning sensor representations with language

"I speak fluent 200 Hz accelerometer and the language model speaks fluent English. Someone still has to translate, and it is going to be a matrix."

A Bilingual AI Agent

The problem is concrete. Suppose a wrist accelerometer window has been encoded into a 512-dimensional vector \(z \in \mathbb{R}^{512}\) by a model from Chapter 20. A language model represents each subword token as, say, a 4096-dimensional vector, and it only knows how to attend over such vectors. The dimensions do not match, the geometry does not match, and nothing tells the LLM that a particular direction in sensor space means "the wrist was shaking." Alignment is the learned map that places sensor content where the language model can find it. Two broad strategies dominate, and they answer a different question: contrastive alignment builds a shared space that both modalities are pulled into, while projection-and-adaptation leaves the LLM fixed and translates sensor vectors into its input token space.

Two spaces, one bridge: the alignment problem

What we are aligning is a sensor embedding \(z = f_\theta(x)\), the output of a sensor encoder over a raw window \(x\), with the text-conditioned representation space of a language model \(g_\phi\). Why this is not automatic: \(f_\theta\) was trained (by forecasting, masked reconstruction, or contrastive pretext tasks) to make signals that look alike sit close together, with no pressure whatsoever to line those directions up with the meaning of English words. The two encoders were optimized in separate universes. Alignment is the extra training stage that couples them, and its supervision almost always comes from paired data: windows of signal that come with a matching piece of text (an activity label, a clinician's note, a maintenance log line, a caption). When you have such pairs at scale, alignment becomes a learnable objective; when you do not, it becomes the central bottleneck of the entire enterprise, which is why Part V returns so often to leakage-safe paired datasets (Chapter 5).

Contrastive alignment: pulling both modalities into one space

The first family builds a shared embedding space with a dual-encoder, contrastive objective, the same machinery you met in Chapter 17 and that CLIP made famous for images and text. How it works: encode a batch of paired signal windows and texts, project each into a common \(d\)-dimensional space, and train so that a window sits closer to its own caption than to any other caption in the batch. With normalized embeddings \(u_i\) (signal) and \(v_i\) (text) and temperature \(\tau\), the symmetric InfoNCE loss is

Why this is attractive: once the space is shared, you get zero-shot classification for free. Encode a new signal, encode a handful of candidate text labels, and pick the nearest. You never trained a classifier; you trained a geometry. Meta's ImageBind extends exactly this idea to inertial measurement unit (IMU) data among six modalities, binding them to a common space anchored on images, and IMU2CLIP aligns wrist and head IMU streams to CLIP's text space so that motion can be retrieved by natural-language query. When to reach for it: retrieval, open-vocabulary tagging, and cross-modal search, where the deliverable is a similarity, not a sentence. Its limit is precisely that it produces a single vector, not language: a shared space can tell you a window is near the phrase "climbing stairs," but it cannot compose a paragraph explaining why, which is where the second family comes in.

Projection and adaptation: sensors as soft tokens for a frozen LLM

The second family keeps a full language model and feeds sensor content into it as if it were extra tokens. What the bridge is here: a small trainable module, the connector, that maps sensor embeddings into the LLM's input embedding space, producing "soft tokens" the model attends over alongside the text prompt. How the connector is built spans a spectrum. At the simple end is a linear or two-layer projection, the design LLaVA popularized for images: one matrix (plus a nonlinearity) turns each sensor vector into one LLM-dimension vector, and you fine-tune only that projection while the language model stays frozen. At the richer end is a cross-attention resampler, the Q-Former of BLIP-2, in which a small set of learned query vectors attend over the sensor encoder's output sequence and distill it into a fixed, short set of tokens regardless of how long the signal was. Systems such as PandaGPT and OneLLM generalize this connector-into-a-frozen-LLM pattern across many modalities at once.

Why this family dominates sensor-language reasoning: because the output is language. Once sensor content is sitting in the prompt as tokens, the LLM's full generative and reasoning ability applies to it, so it can caption, answer, and explain, not merely score similarity. When the linear projector suffices versus when you need a resampler: a projector is cheap and shockingly effective when one window maps to one token and the signal is short; a resampler earns its complexity when the sensor encoder emits a long token sequence (a multi-minute, multi-channel stream) that must be compressed to a few tokens so it does not swamp the context window. The listing below trains the minimal version of this bridge, a linear projector, against the contrastive objective, so both families appear in one place.

import torch, torch.nn.functional as F

class SensorToTextBridge(torch.nn.Module):
    """Project frozen sensor embeddings into a text-embedding space."""
    def __init__(self, d_sensor=512, d_text=4096):
        super().__init__()
        self.proj = torch.nn.Sequential(
            torch.nn.Linear(d_sensor, d_text), torch.nn.GELU(),
            torch.nn.Linear(d_text, d_text))     # the only trainable weights

    def forward(self, z):                          # z: (batch, d_sensor)
        return self.proj(z)                        # soft tokens: (batch, d_text)

def info_nce(u, v, tau=0.07):
    u, v = F.normalize(u, dim=-1), F.normalize(v, dim=-1)
    logits = (u @ v.t()) / tau                     # (batch, batch) similarity
    target = torch.arange(u.size(0), device=u.device)
    return 0.5 * (F.cross_entropy(logits, target) +
                  F.cross_entropy(logits.t(), target))

bridge = SensorToTextBridge()
z_sensor = torch.randn(16, 512)                    # from a frozen Ch.20 encoder
t_text   = torch.randn(16, 4096)                   # frozen text embeddings of captions
loss = info_nce(bridge(z_sensor), t_text)          # align, LLM stays frozen
loss.backward()
print(f"alignment loss = {loss.item():.3f}  |  trainable params only in the bridge")

Listing 1 makes the economy of alignment visible: the sensor encoder is frozen, the text side is frozen, and the entire coupling lives in a handful of megabytes of projection weights. That is why a lab can align a new sensor to an existing language model in hours rather than pretraining from scratch. In a full projection-and-adaptation system you would replace the InfoNCE target with a next-token loss on captions or answers, so the LLM learns to generate from the soft tokens rather than merely sit near them, but the trainable surface stays just as small.

from transformers import AutoModel
# A vision-language model IS the projector + frozen LLM pattern, reusable for sensors
model = AutoModel.from_pretrained("llava-hf/llava-1.5-7b-hf")
soft_tokens = model.multi_modal_projector(sensor_features)   # encoder features -> LLM space

What alignment must not throw away

A bridge that compiles is not a bridge that preserves meaning. Three properties of the sensor signal are easy to lose in alignment and expensive to recover. Physical semantics: the mapping must keep the direction that means "atrial fibrillation" distinct from the one that means "motion artifact," which demands paired data where those are actually distinguished, not conflated under one label. Scale and units: a language model has no innate sense that 2 g of acceleration differs from 0.2 g; if the sensor encoder normalized that away, alignment cannot invent it, so magnitude that matters clinically or mechanically must survive into the embedding (a recurring theme from Chapter 2). Temporal structure: collapsing a long window to one token discards when things happened, so any reasoning about order or duration (the point of Section 21.4) needs a connector that emits a short sequence of time-ordered tokens, not a single averaged vector. Naming these now matters because the failure they cause, an LLM confidently describing signal content that its aligned tokens never actually encoded, is exactly the grounding-and-hallucination problem Section 21.6 is devoted to.

What's Next

In Section 21.2, we put the projection-and-adaptation bridge to work on its first real task: turning aligned sensor tokens into fluent, faithful natural-language descriptions of a signal, and confronting the gap between a caption that reads well and a caption that is actually true to the waveform.