"A number with no units, no clock, and no sensor ID is not data. It is a rumor. I can fit a model to a rumor, and it will be exactly as trustworthy as one."
A Fastidious AI Agent
The big picture
The previous six sections treated the sample as if it arrived on your desk labeled and ready. It does not. Between the transducer and the array your model consumes sits an acquisition stack: a chain of buses, transports, and drivers that moves raw counts off silicon, plus a body of metadata that records what those counts mean. Get the protocol wrong and you drop samples or scramble their order; omit the metadata and you ship a stream that is numerically intact yet scientifically meaningless. This section is about the layer where measurement becomes a durable, self-describing record. It is unglamorous plumbing, and it is where most real sensor projects quietly fail: not in the model, but in a dataset whose sample rate was guessed, whose units were assumed, and whose provenance nobody wrote down.
This section closes Chapter 3 by tying the acquisition machinery to the timing discipline of Section 3.3 and the framing of Section 3.6. It leans on the calibration and units vocabulary from Chapter 2, and it is the direct on-ramp to the dataset engineering of Chapter 5. Where Chapter 5 asks how to build leakage-safe datasets, this section supplies the raw material: the fields you must capture at acquisition time, because they cannot be reconstructed afterward.
The acquisition stack: from counts to a timestamped record
What it is. Getting a sample from a transducer to storage crosses at least two protocol layers. At the bus level, a microcontroller pulls raw registers over a wired interface: I2C and SPI for on-board chips (an IMU or a pressure sensor), UART or CAN in vehicles, and analog lines into an ADC for everything else. At the transport level, framed samples travel to a host or the cloud over MQTT, gRPC, WebSocket, a ROS topic, or Lab Streaming Layer. Each layer imposes its own bandwidth ceiling, its own framing, and its own failure mode.
Why the choice matters. The protocol decides three things a model inherits. First, ordering and loss: a lossy transport (UDP, BLE notifications) can reorder or drop frames, so you cannot assume sample \(n+1\) followed sample \(n\) in time (revisit Section 3.4 on packet loss). Second, where the timestamp is stamped: a clock applied at the sensor MCU is worth far more than one applied when a buffer finally reaches the host, because the intervening queue delay is variable and unknown. Third, throughput headroom: an I2C bus at 400 kHz cannot sustain a 4 kHz burst across eight chips, so the driver silently decimates and your effective rate is a fiction. The competent answer is to timestamp as early as physically possible, prefer transports that expose sequence numbers, and measure the real delivered rate rather than trusting the configured one.
Push versus pull. Two acquisition disciplines dominate. In polling, the host reads on a timer, which is simple but couples your timeline to the host scheduler's jitter. In interrupt or FIFO mode, the sensor buffers samples on-chip and raises a data-ready line, so timing is governed by the sensor's own crystal and the host merely drains a queue. FIFO acquisition is almost always the right default for anything above a few hundred hertz, because it decouples the sample clock from the noisy host and lets you batch transfers to save power, a tradeoff that returns in force in Chapter 59.
Metadata: the fields that make a stream self-describing
What it is. Metadata is the set of facts required to interpret a sample that are not themselves samples. A defensible sensor record carries, at minimum: a device and channel identifier, the nominal and measured sample rate, physical units and the scale factor from raw counts to those units, the sensor range and configured gain, the timestamp source and its epoch and timezone, the firmware and calibration versions, and the sensor's placement or mounting. None of these are optional decoration. Each answers a question a downstream model or auditor will eventually ask, and each is cheap to record now and impossible to recover later.
Why it is load-bearing. Consider what breaks without each field. No units, and a fusion stage adds acceleration in \(g\) to acceleration in \(\text{m/s}^2\). No measured rate, and your spectral analysis in Chapter 7 places every peak at the wrong frequency. No calibration version, and a firmware update that changed the gain silently splits your dataset into two incompatible populations. No device ID, and you cannot split train and test by device, the single most important guard against the leakage that Chapter 5 is built around. Metadata is the join key between a raw stream and its physical meaning.
Key insight
Record the metadata that cannot be recovered from the samples, and record it at the moment of acquisition. A sample rate can sometimes be re-estimated from timestamps; a sensor's firmware version, mounting orientation, and coordinate convention cannot be inferred from the numbers at all. The test for whether a field belongs in metadata is simple: if a stranger handed only the raw array could reconstruct it, it is derivable; if they could not, it must be captured now or it is gone. Treat acquisition metadata as write-once provenance, not as a configuration file you might edit later.
Schemas and standards: agreeing on the record's shape
Why standards exist. Every lab that invents its own metadata layout invents its own way to be incompatible with everyone else, including its future self. Community schemas encode the hard-won list of required fields so you do not rediscover it after losing a dataset. In biosignals, EDF+ standardized channel labels, per-channel scaling, and annotations for clinical recordings; the neuroimaging world formalized folder-level provenance as BIDS. For self-describing arrays, HDF5 and its cloud-native cousin Zarr attach named dimensions and arbitrary attributes directly to the data, so units and sample rate travel inside the file rather than in a fragile sidecar. For sensor descriptions on the web, W3C's SOSA/SSN ontology and OGC SensorML give a vocabulary for observations, procedures, and platforms. Robotics leans on ROS bag files and message definitions to keep a timestamped, typed record of every topic.
The FAIR frame. The unifying principle is that data should be Findable, Accessible, Interoperable, and Reusable. In practice FAIR reduces to a discipline you can apply on any project: a stable identifier per recording, a documented schema, units and vocabularies that other tools recognize, and enough provenance that a result can be reproduced. You do not need a heavyweight ontology on day one; you need to pick one self-describing container and never write a bare CSV of unlabeled columns again.
In practice: the automotive fleet whose two trucks disagreed
A logistics company logged wheel-speed and IMU data over the vehicle CAN bus across a fleet to train a road-surface classifier. The pilot model scored beautifully in validation and then behaved erratically on new trucks. The cause was buried in missing metadata. Half the fleet ran a firmware revision that reported yaw rate in degrees per second; the other half, after an over-the-air update, reported radians per second, and nobody logged the firmware version alongside the samples. The two conventions differ by a factor of about 57, so the model had effectively learned to separate trucks by their units and called it "road surface." Worse, because validation split randomly rather than by vehicle, both unit conventions appeared in training and test, hiding the defect until deployment on a truck the split had never isolated. The fix was not a better model. It was a one-line addition to the CAN logger that stamped the firmware version, units, and vehicle ID into every record, plus a device-wise split. The classifier's honest accuracy dropped, then became real.
The code below encodes the discipline as a small guard: an ingestion gate that refuses any batch missing the fields required to interpret it, and that flags when the measured rate drifts from the nominal one. Wiring this in front of storage turns a whole class of silent dataset corruption into a loud, early failure.
import numpy as np
REQUIRED = ("device_id", "channel", "units", "scale",
"nominal_rate_hz", "t0_unix", "timezone",
"firmware", "calibration_id")
def ingest(samples_raw, timestamps, meta, rate_tol=0.02):
missing = [k for k in REQUIRED if k not in meta]
if missing:
raise ValueError(f"refusing batch: missing metadata {missing}")
# Verify the delivered rate against the claim; a drift here means
# dropped samples, a wrong config, or a lying driver.
dt = np.diff(timestamps)
measured = 1.0 / np.median(dt)
if abs(measured - meta["nominal_rate_hz"]) / meta["nominal_rate_hz"] > rate_tol:
raise ValueError(f"rate mismatch: claimed {meta['nominal_rate_hz']} Hz, "
f"measured {measured:.1f} Hz")
physical = samples_raw.astype(np.float64) * meta["scale"] # counts -> units
return {"values": physical, "t": timestamps, "meta": meta,
"measured_rate_hz": float(measured)}
meta = dict(device_id="imu-0421", channel="gyro_z", units="rad/s", scale=1.526e-4,
nominal_rate_hz=200.0, t0_unix=1_752_000_000.0, timezone="UTC",
firmware="2.3.1", calibration_id="cal-2026-05-tumble")
raw = np.random.randint(-2000, 2000, size=1000)
t = meta["t0_unix"] + np.arange(1000) / 200.0
print(ingest(raw, t, meta)["measured_rate_hz"])scale field converts raw gyro counts to rad/s, and the rate check catches the dropped-sample and wrong-config failures from Section 3.4 before they reach storage. This function is the enforcement point referenced by the metadata discipline above.The right tool: self-describing arrays instead of a sidecar zoo
Hand-managing a parallel JSON sidecar for every array (keeping units, rates, coordinates, and axis labels in sync with the data through every slice and resample) is roughly eighty lines of bookkeeping that drifts out of date the first time someone edits one file and not the other. An xarray dataset attaches named dimensions, coordinates, and attributes to the array itself, so the metadata rides inside the object and survives slicing:
import numpy as np, xarray as xr
da = xr.DataArray(
np.random.randn(1000, 3), dims=("time", "axis"),
coords={"time": np.arange(1000) / 200.0, "axis": ["x", "y", "z"]},
attrs={"units": "rad/s", "device_id": "imu-0421",
"nominal_rate_hz": 200.0, "firmware": "2.3.1"})
da.to_netcdf("gyro.nc") # units + coords + attrs travel with the data
window = da.sel(time=slice(0.0, 1.0)) # metadata preserved through the slice
print(window.attrs["units"], window.sizes)xarray: named axes, a real time coordinate, and physical attributes stored with the array and written to a single netCDF/HDF5 file. Eighty lines of sidecar synchronization collapse to a constructor call, and the units cannot silently detach from the numbers.Exercise
Harden the ingestion gate into something you would trust in a fleet:
- Extend
ingestto also reject a batch whose timestamps are non-monotonic (out-of-order delivery) and to report the fraction of inter-sample gaps that exceed twice the nominal period, a proxy for packet loss. - Add a metadata field
coordinate_frameand write a check that two IMU streams claiming to fuse share the same frame, refusing the fusion otherwise. Explain which single character of a wrong answer this would have caught in the automotive story. - Serialize an accepted batch to both a bare CSV and an
xarraynetCDF file. Hand each to a colleague with no context and note which fields they can recover from each.
Hint
For part one, np.all(np.diff(timestamps) > 0) tests monotonicity and the gap fraction is np.mean(np.diff(timestamps) > 2 / rate). For part three, the CSV strips units, rate, and provenance the moment it is written; that asymmetry is the whole argument for self-describing containers.
Self-check
- Why is a timestamp applied at the sensor MCU generally more trustworthy than one applied when the buffer reaches the host?
- Name three metadata fields that cannot be reconstructed from the raw samples alone, and one that sometimes can.
- A dataset trains and tests by random split and reaches suspiciously high accuracy. Which single metadata field would let you re-split to expose device-level leakage, and why does that field belong in the acquisition record rather than added later?
Lab 3
build a streaming ingestion pipeline with windowing, missing-data markers, and cross-sensor time alignment.
Bibliography
Sensor description and metadata standards
The web standard vocabulary for describing sensors, observations, procedures, and platforms; the reference model when your metadata must interoperate across organizations.
A rich schema for sensor systems and processing chains, widely used in geospatial and environmental sensor networks where provenance of the measurement process is mandatory.
The de facto container for clinical biosignals, standardizing per-channel labels, physical scaling, and annotations; the model for how to make a physiological recording self-describing.
Self-describing data formats
The HDF Group (1998-present). Hierarchical Data Format version 5 (HDF5).
The workhorse binary container that stores named datasets with attached attributes, letting units and sample rates live inside the file; the substrate under netCDF, NWB, and many sensor archives.
Zarr Developers (2024). Zarr: chunked, compressed, N-dimensional arrays.
Cloud-native self-describing arrays with per-chunk compression and JSON attribute metadata; the format of choice for large sensor archives read in parallel from object storage.
A concrete, adopted example of folder-level provenance and naming conventions; a template for how a community turns metadata discipline into a shareable standard.
Provenance, documentation, and reproducibility
The Findable, Accessible, Interoperable, Reusable principles that frame why acquisition metadata is not optional; the citation behind most modern data-management mandates.
Argues that every dataset should ship a datasheet documenting how it was collected and its intended use; the machine-learning analogue of acquisition metadata, and a direct feed into Chapter 5.
Kothe, C., et al. (2014-present). Lab Streaming Layer (LSL).
A transport that unifies time-series streams from many devices with a common clock and per-stream metadata header; a practical reference design for real-time multi-sensor acquisition.
What's Next
In Chapter 4, we leave the mechanics of getting clean, labeled, timestamped samples and start reasoning about what they tell us. The metadata gate here decides whether a number is admissible; the probability toolkit next decides what to believe once it is admitted, giving us estimators, priors, and the aleatoric-versus-epistemic split that lets every later chapter attach honest uncertainty to a sensor reading rather than a bare point estimate.