A taxonomy of modalities and task families

"First I sorted the sensors by what they measure, then by how fast, and then I had a spreadsheet, and a spreadsheet I can finally understand."

A Taxonomically Inclined AI Agent

How can the same twenty lines of code that flag a failing bearing also flag an abnormal heartbeat? The answer is that both sit at the same address on a two-axis map, and this section builds that map using only the four quantities introduced in Section 1.3: the raw signal \(x(t)\), the hidden world state \(s\), the discrete event \(e\), and the action \(a\). If those symbols are unfamiliar, read that section first; everything here is a way of organizing them. Deployment constraints such as latency, power, and privacy belong to Section 1.6, and the pipeline stages that turn \(x\) into \(a\) belong to Section 1.5. This section answers only what kind of thing a sensor is and what kind of question we pose to it.

Four axes that pin down any modality

A modality is not defined by its brand or its bus. It is defined by where it sits along four axes, and those four coordinates predict almost everything about how you will process it. Figure 1.4.1 illustrates the four modality axes as a sensor feature vector.

Transduced quantity: what physics is being measured. Every sensor is a function \(x = h(s) + \eta\) that maps some hidden state \(s\) to a signal through a response \(h(\cdot)\) with noise \(\eta\). The first question is what \(s\) actually is. An accelerometer transduces proper acceleration; a microphone transduces air pressure; a photodiode transduces irradiance; an electrocardiogram (ECG) electrode transduces the body-surface potential of cardiac depolarization. This choice fixes the noise physics you will fight in Chapter 2. Thermal noise, shot noise, and motion artifact are not interchangeable, and knowing the transduced quantity tells you which one dominates.

Sampling rate: how fast the world is queried. A signal sampled at rate \(f_s\) can only faithfully represent content below \(f_s/2\); above that, energy folds back as aliasing (the sampling theorem, developed in Chapter 3). Human activity lives comfortably at tens of hertz, so wrist inertial sensors run at 25 to 100 Hz. Audio needs kilohertz. An ECG at 250 to 500 Hz resolves the sharp QRS complex (the tall, narrow spike in each heartbeat that marks the ventricles contracting); an electroencephalogram (EEG) event of interest may demand more. Sampling rate sets your window length, your memory budget, and whether a Fourier view even makes sense.

Aliasing is the specific failure that bound guards against. It is the irreversible corruption that appears when signal energy above \(f_s/2\) reaches the converter anyway, and it matters because no later filtering can undo it once the samples are stored. The mechanism folds, or mirrors, the too-fast content back down into the low frequencies. There it masquerades as a genuine low-frequency component, and you can no longer separate it from the real thing. When the sensor's rate is fixed and you cannot raise \(f_s\), insert an analog anti-alias low-pass filter before the converter. When you control the rate, sampling faster than twice the highest frequency of interest is the cleaner remedy.

From time to space: shape and structure

Rate settles the temporal grain of a signal; the third axis turns from time to space. Dimensionality: spatial extent and channel count. A thermistor is a scalar time series (zero spatial dimensions, one channel). A tri-axial accelerometer is three channels of a point. A camera frame is a 2D grid with three color channels. A lidar sweep is an unordered set of 3D points. This axis decides your architecture: 1D convolutions and recurrent models for streams, 2D convolutions or vision transformers for grids, and point-set or graph networks for the geometry covered in Chapter 40.

Output structure: how samples are organized in time. Some sensors emit uniformly clocked frames. Others, such as event cameras, emit asynchronous, sparse spikes only when something changes, so there is no fixed \(f_s\) at all. Structure determines whether you can stack a tensor of shape (time, channels) or must reason over irregular timestamps. In short: pin a sensor to four coordinates and its question to one of six task families, and you stop writing seventy-two device manuals and start reusing one recipe per cell.

Six task families that recur everywhere

The second axis is the question. Across every modality, the inference targets collapse into six recurring families, each defined by the type of its output rather than by the domain.

• Classification. Map a window to one label from a fixed set: \(x[n] \mapsto c \in \{1,\dots,C\}\). "Walking vs running", "arrhythmia vs sinus rhythm".
• Regression and forecasting. Map to a continuous quantity, either now (estimate current heart rate from photoplethysmography (PPG)) or ahead in time (forecast \(x_{k+1{:}k+H}\) from \(x_{1{:}k}\)). The forecasting variant is the province of Chapter 19.
• Detection. Localize discrete events \(e\) in time (or space): find when a footstep, a keyword, or a machine fault onset occurs, not just whether the window contains one.
• State estimation. Recover a continuous latent \(\hat{s}_t\) that evolves over time, typically with a dynamics model: orientation, position, battery charge. This is the Kalman and Bayesian-filtering territory of Part III.
• Segmentation. Assign a label to every sample (temporal) or every pixel or point (spatial), partitioning the signal into contiguous regions.
• Anomaly detection. Flag samples that depart from a learned notion of normal, usually without labeled examples of the failure. Central to condition monitoring.

These families are not exclusive. Human activity recognition, treated in depth in Chapter 26, is windowed classification when you want a label per two-second clip, but temporal segmentation when you want exact transition boundaries. Naming the family is the first design decision, because it fixes the loss function, the metric, and the label format long before you pick a model.

The modality-by-task matrix

Get this cross wrong and device teams often quietly rebuild the same detector from scratch, several times over, each one blind to the others who already solved it. Cross the two axes and you get the organizing artifact of this section: a grid whose rows are modalities and whose columns are task families. Each cell names the canonical question people actually ask of that pairing. The matrix in the table below is a small slice; the point is not to be exhaustive but to show that the columns are the same no matter how alien the rows look to each other.

Read the table by columns and the recurrence stands out. The "anomaly detection" column poses one statistical question, is this point off the learned manifold of normal (the manifold being the low-dimensional surface that healthy data traces out inside the space of all possible signals), whether the point is a gait cycle, a heartbeat, or a bearing spectrum. A method invented for one row therefore transfers to the next, and a single foundation model can increasingly serve a whole column, dozens of physically unrelated instruments answered by one frozen backbone (the term is made concrete in the Research Frontier note just below). Read the table by rows and you get the planning template for Lab 1: place each dataset you survey in exactly one row-and-column cell, then annotate three cross-cutting properties the matrix alone does not capture.

Three cross-cutting properties the matrix does not show

A cell tells you the shape of the question. It does not tell you how hard the data will be to work with. Three additional properties, which you will record for every dataset in Lab 1, cut across all cells.

Noise source. The \(\eta\) in \(x = h(s) + \eta\) is not generic. Wearable inertial data is dominated by motion artifact and loose-contact dropouts; ECG fights baseline wander and mains interference; lidar suffers from reflectivity and range dropout. Knowing the dominant noise source picks your cleaning strategy in Part II.

Label cost. Some labels are nearly free (a machine's own controller logs when it faulted). Others require a clinician-adjudicated gold standard at real per-sample expense. High label cost is precisely what pushes a project toward self-supervised (learning from unlabeled data by predicting one part of a signal from another) and foundation-model approaches, and it is the reason anomaly detection, which needs no fault labels, is so often the first thing to ship.

Deployment constraint. A model that must run on a coin-cell wearable faces different limits than one on a vehicle compute stack. We only tag this property here; Section 1.6 promotes latency, bandwidth, power, cost, and privacy to first-class citizens.

Reading a dataset into the taxonomy, in code

With those three cross-cutting properties noted for later, the four core axes are the part a program can compute directly. The taxonomy is meant to be operational, not decorative. The short script below encodes each modality as its four-axis coordinate and prints the compact "sensor feature vector" you will build a spreadsheet of in Lab 1. It is the seed of that survey: given raw metadata, it classifies the sensor before a single sample is processed.

import pandas as pd

# Each row is one modality described by the four taxonomy axes.
sensors = pd.DataFrame([
    #  name,            quantity,        f_s_hz, channels, spatial_dims, output
    ["wrist_imu",      "acceleration",     50,       6,        0,     "frames"],
    ["ecg_lead",       "biopotential",    360,       1,        0,     "frames"],
    ["microphone",     "air_pressure",  16000,       1,        0,     "frames"],
    ["lidar",          "range",            10,       1,        3,     "point_set"],
    ["event_camera",   "log_irradiance",   -1,       1,        2,     "async_events"],
], columns=["name", "quantity", "f_s_hz", "channels", "spatial_dims", "output"])

def nyquist(f_s):
    return "asynchronous" if f_s < 0 else f"{f_s / 2:.0f} Hz usable band"

sensors["max_content"] = sensors["f_s_hz"].apply(nyquist)
sensors["shape_class"] = sensors["spatial_dims"].map({0: "stream", 2: "grid", 3: "geometry"})

print(sensors[["name", "quantity", "max_content", "shape_class", "output"]].to_string(index=False))

Notice what the code makes visible. The event camera has no meaningful \(f_s/2\) because it is not clocked at all, so its usable band prints as asynchronous. Spatially it is still a 2D grid, the same shape class as an ordinary frame camera, yet its output structure is async_events rather than uniform frames. That contrast is why "output structure" earns its place as a full axis rather than a footnote to sampling rate: it is the coordinate that separates an event camera from a frame camera that matches it on every other number.

import soundfile as sf
x, f_s = sf.read("machine.wav")          # samples, sampling rate
channels = 1 if x.ndim == 1 else x.shape[1]
print(f_s, channels, x.shape[0] / f_s, "seconds")

The figure below plots the two axes as a plane. Any sensing project is a single point on it: pick a modality row and a task column to locate your problem in the territory this book spends fourteen parts mapping.

What's Next

In Section 1.5, we stop asking what kind of thing and start tracing how it flows: the sensing chain that carries a raw signal \(x(t)\) through conditioning, representation, and inference to an action \(a\). The taxonomy you built here becomes the labels on the chain's input; the next section connects those inputs to the six verbs that structure the whole book. A fuller catalog of the datasets referenced throughout, with their axes pre-tabulated, lives in Appendix E: Datasets and Benchmarks.