Sensor question answering (SensorQA, SensorBench, OpenSQA)

"You handed me eleven days of accelerometer counts and asked whether you had been active enough. Active enough for what, by whose threshold, over which of the eleven days? The signal is precise; your question is not."

A Literal-Minded AI Agent

This section assumes the activity-recognition vocabulary of Chapter 26 (the labels a wearable QA system reasons over), the sampling-and-time discipline of Chapter 3 (a question about "last Tuesday" is meaningless without a trustworthy clock), and the leakage-safe splitting rules of Chapter 5, which turn out to decide whether a QA benchmark measures reasoning or memorization.

What a sensor question actually asks

Before comparing benchmarks, fix a taxonomy, because the single most important property of a sensor-QA dataset is the distribution of question types it samples. A useful ladder, in rough order of difficulty:

• Retrieval / lookup: "What was I doing at 3pm?" One span, one label. Solvable by the classifier alone.
• Aggregation: "How many hours did I walk today?" Requires summing over segments, so the model must first segment, then count, then convert units.
• Comparison: "Did I sleep more this week than last?" Two aggregates and a relation.
• Temporal / trend: "Has my sedentary time been increasing?" A slope over aggregates, sensitive to the window the question implies.
• Causal / explanatory: "Why did my heart rate spike at noon?" The model must cross-reference channels (activity, motion) and produce a justified narrative, the hardest and least automatically scorable class.

Every real question maps to one or more rungs of this ladder, and performance collapses predictably as you climb it. A model can nail retrieval and still be useless at comparison, because comparison composes two error-prone aggregates and the errors multiply. This is why "accuracy on sensor QA" is a near-meaningless headline: it averages over question types whose difficulty spans an order of magnitude.

Three benchmarks, three design choices

SensorQA is the human-grounded reference point. It pairs long-term daily-activity time series (the multi-day accelerometer-plus-context streams of Chapter 26) with roughly 5.6K question-answer pairs written and answered by human annotators reading visualized activity timelines. Because humans wrote the questions, the distribution is realistic: it leans on aggregation, comparison, and duration over long horizons rather than instant lookup. Its central finding is practical and sobering: models small enough to run on a phone lag well behind large server models, so the systems you would actually deploy at the edge are the ones that struggle most on natural questions. SensorQA measures the gap you fight in production.

SensorBench asks a different question: not "can a model answer a daily-life query" but "can a large language model process raw sensor signals at all". It presents an LLM with numerical sensor data and a battery of tasks ordered by difficulty, from property extraction (find the peak, the mean, the dominant period) up to composed multi-step processing, while systematically varying the prompting strategy. Its lesson mirrors the taxonomy above: models handle atomic operations far better than compositions, and prompt scaffolding (decomposition, explicit computation) helps most exactly where naive prompting fails. SensorBench is a diagnostic of raw-signal competence, upstream of any natural-language interface.

OpenSQA pushes toward open-ended, instruction-style answering. Where SensorQA's human set is small and closed, an open-ended corpus is generated at scale to instruction-tune a sensor-language model, teaching it to produce free-form explanatory answers rather than pick from a fixed set. It trades the guaranteed correctness of human labels for the volume needed to fit a model, and it foregrounds the hardest rung, explanations, where scoring is open research. Together the three cover the pipeline: OpenSQA-style data to train, SensorBench to probe raw-signal skill, SensorQA to check natural-question performance under an honest edge-versus-cloud comparison.

How you score a free-form answer

Evaluation is where sensor QA quietly gets hard. For a categorical lookup, exact match or token F1 (borrowed from reading-comprehension QA) is fine. For aggregation and comparison the answer is a number with units, and exact match is nonsense: "3.9 hours" against a gold "4 hours" is essentially correct and should not score zero. The workable convention is a tolerance-banded numeric match: parse the quantity, then accept it if it falls within a relative tolerance of the gold value. For a predicted quantity \(\hat{y}\) and gold \(y\),

with a small \(\epsilon\) to guard the zero-gold case and a tolerance \(\delta\) (often 0.1 to 0.2) chosen to match how precisely the question can even be answered from the sensor. The code below implements exactly this split: categorical answers get normalized exact match, numeric answers get the tolerance band. Reporting one blended accuracy hides the story, so it also breaks the score out by question type, which is the number that actually guides model improvement.

import re

def parse_qty(ans):
    m = re.search(r"[-+]?\d*\.?\d+", str(ans))     # first number in the string
    return float(m.group()) if m else None

def score_answer(pred, gold, kind, delta=0.15, eps=1e-6):
    if kind == "categorical":
        norm = lambda s: re.sub(r"\s+", " ", str(s).strip().lower())
        return float(norm(pred) == norm(gold))
    yhat, y = parse_qty(pred), parse_qty(gold)       # numeric: tolerance band
    if yhat is None or y is None:
        return 0.0
    return float(abs(yhat - y) / max(abs(y), eps) <= delta)

qa = [
    ("walking",        "Walking",        "categorical"),
    ("about 3.9 hours","4 hours",        "numeric"),
    ("5.5 hours",      "4 hours",        "numeric"),
]
from collections import defaultdict
by_kind = defaultdict(list)
for pred, gold, kind in qa:
    by_kind[kind].append(score_answer(pred, gold, kind))
for kind, s in by_kind.items():
    print(f"{kind:12s} acc = {sum(s)/len(s):.2f}")

Building your own set without poisoning it

You will usually need a domain-specific QA set, and the failure modes are the book's earlier ones, sharpened. First, split by subject and by time: if windows from one wearer (or worse, adjacent windows from one recording) land in both train and test, a model that merely memorized that person's routine scores like a reasoner. This is the Chapter 5 discipline applied to the question, not just the signal. Second, balance the question-type ladder deliberately; a set scraped from convenient templates drifts toward lookup and flatters classifiers. Third, because comparison and trend answers carry compounding uncertainty, a deployed QA system should report calibrated confidence rather than a bare number, which is exactly the conformal machinery of Chapter 18.

What's Next

In Section 21.4, we move from measuring whether a model can answer sensor questions to the mechanisms that let it reason over raw streams in the first place: LLaSA, SensorLLM, HARGPT, and LLMSense, and the architectural choices (tokenizing signals, aligning them to language, prompting for step-by-step inference) that turn a language model into a sensor reasoner.