Formats for telemetry, waveforms, event streams, and point clouds

"They handed me a folder of float32 arrays with no sample rate, no units, and no timezone. I did not have a dataset. I had a rumor."

A Disillusioned AI Agent

Four shapes, not one

Almost every sensor stream you will meet falls into one of four structural families, distinguished by three questions: are the samples uniformly spaced in time, is the data dense or sparse, and does each record hold a fixed or variable number of values? Get the family right and the storage decision nearly makes itself.

Telemetry is low-rate, multivariate, timestamped scalars: engine RPM, battery voltage, GPS latitude, cabin temperature, each a named channel sampled from a fraction of a hertz up to roughly a hundred. It is naturally a wide table, one row per timestamp, one column per channel. Waveforms are dense, uniformly sampled continuous signals: audio at 44.1 kHz, an ECG trace at 500 Hz, a bearing accelerometer at 25.6 kHz. Here the sample rate is not data to store per row; it is a single number that, together with a start time, lets you reconstruct every timestamp implicitly. Event streams are asynchronous: things happen at irregular instants and you record a tuple when they do. A CAN message, a syslog line, a keystroke, or an event-camera pixel firing \((x, y, \text{polarity}, t)\) all share the trait that time between records is itself information. Point clouds are unordered sets: a single lidar sweep is a variable-size bag of points, each carrying \((x, y, z)\) plus intensity and often a per-point timestamp, with no meaningful row order and a different cardinality every frame.

Table 5.1.1 lays these four side by side. The reason this taxonomy matters is that each shape breaks a different default. Telemetry punishes row-major storage when you only need two of two hundred channels. Waveforms punish any format that stores a timestamp per sample. Event streams punish fixed-rate assumptions. Point clouds punish anything that expects a rectangular array. Choosing the layout is choosing which of these mistakes you refuse to make.

Rows, columns, and chunks: why layout is destiny

The deepest split in sensor storage is row-major versus columnar. A row-major file (CSV, JSON lines, a naive database table) stores all fields of record 1, then all fields of record 2, and so on. A columnar file (Parquet, Arrow, ORC) stores all values of channel A contiguously, then all of channel B. For telemetry the columnar layout is transformative for three concrete reasons. First, selective reads: an analyst who wants only rpm and oil_temp from a two-hundred-channel truck log reads two columns off disk and skips the rest, instead of parsing every byte of every row. Second, compression: values within one channel are similar (temperature changes slowly), so a column of them compresses far better than interleaved heterogeneous rows; run-length and dictionary encoding routinely shrink telemetry by five to twenty times. Third, predicate pushdown: column statistics stored per block let a reader skip entire chunks whose value range cannot match a filter, so "give me the rows where RPM exceeded 6000" touches only the blocks that could contain them.

Waveforms want a different discipline. A dense uniform signal has no per-sample metadata worth storing, so the right structure is a chunked array: a long contiguous block of one dtype, split into fixed-size chunks along the time axis, with the sample rate, units, dtype, start time, and device identity held once as attributes. HDF5 and Zarr do exactly this; the venerable WAV and biosignal EDF (European Data Format) containers do a domain-specific version of the same idea. Chunking is what makes a multi-hour recording streamable: you read the ten-second window you need without materializing the whole file, which is precisely the access pattern Section 5.2 builds windows on. The recurring principle across both cases is that timestamps for a uniform signal are computed, never stored: sample \(n\) sits at time \(t_0 + n/f_s\), so persisting a timestamp column would waste space and, worse, let rounding error desynchronize the axis. Getting \(t_0\) and \(f_s\) right ties directly back to the synchronization discipline of Chapter 3.

Event streams and point clouds: the irregular families

Event streams and point clouds resist the tidy rectangle, and pretending otherwise is where naive pipelines quietly corrupt data. An event stream carries its meaning in the gaps: a burst of CAN messages during hard braking and silence during cruise are both signal. Store events as a row log where the timestamp is a first-class column, never resample them onto a fixed grid at ingestion time (that decision belongs downstream and destroys information if baked in), and keep them monotonically ordered by time. Robotics leans on MCAP and the older rosbag containers precisely because they log heterogeneous, asynchronous messages with per-message timestamps and typed schemas; neuromorphic vision uses AEDAT and EVT to pack millions of \((x, y, p, t)\) tuples per second, the format that Chapter 46 builds event-camera perception on. A columnar log such as Parquet also serves general event data well, since the timestamp and payload columns compress and filter efficiently.

A point cloud is a set, and sets have two properties that break array-shaped storage: variable cardinality (each sweep has a different point count) and permutation invariance (no canonical order). You cannot stack frames into one rectangular tensor without padding, and you must never let a tool assume row order carries meaning. Automotive lidar pipelines commonly store each frame as a packed little-endian binary blob of \(N \times 4\) float32 values \((x, y, z, \text{intensity})\) with \(N\) recorded in a sidecar index; geospatial workflows use LAS and its compressed cousin LAZ; general 3D tooling uses PCD and PLY. Whatever the container, the metadata that must ride along is the sensor pose, the coordinate frame, and the per-point time offset, because a sweep is not instantaneous and the vehicle moved while it scanned. The models that consume these sets, and the reason their architectures must be permutation-invariant, are the subject of Chapter 42.

Self-describing or worthless

Across all four shapes, one property separates a dataset from a rumor: the file must carry enough metadata to reconstruct physical meaning without external knowledge. The minimum viable descriptor is the tuple in the code below, and its absence is the single most common way sensor data arrives broken. A bare float32 array cannot be interpreted without its dtype and endianness (is that little-endian float or big-endian int?), its sample rate and start time (or the time axis is unknowable), its units and scale (millivolts or volts, raw ADC counts or calibrated), its device and session identity (or you cannot later build the device-disjoint splits that Section 5.4 requires), and its timezone or an unambiguous epoch (or midnight is a coin flip). Listing 5.1.1 writes a waveform to Parquet with exactly this descriptor attached as schema metadata, then reads it back and reconstructs the time axis from \(f_s\) and \(t_0\) alone.

import numpy as np, pyarrow as pa, pyarrow.parquet as pq

fs, t0 = 25600.0, 1_726_000_000.0          # Hz, and a UTC epoch second
sig = np.random.randn(256_000).astype(np.float32)   # ~10 s of vibration

# Attach the self-describing descriptor as column-schema metadata.
table = pa.table({"accel_axial": sig})
table = table.replace_schema_metadata({
    b"sample_rate_hz": b"25600.0",
    b"start_epoch_utc": b"1726000000.0",
    b"units": b"m/s^2",
    b"device_id": b"turbine-044-gearbox",
    b"session_id": b"2026-07-14T09:00Z",
})
pq.write_table(table, "vib.parquet", compression="zstd")

# A reader anywhere reconstructs the time axis with no side channel.
back = pq.read_table("vib.parquet")
meta = back.schema.metadata
fs_r = float(meta[b"sample_rate_hz"])
n    = back.num_rows
t    = float(meta[b"start_epoch_utc"]) + np.arange(n) / fs_r   # implicit, exact

As Listing 5.1.1 shows, carrying device_id and session_id is not bookkeeping pedantry. Those fields are the hooks that make leakage-safe splitting possible at all; drop them at ingestion and no downstream care can recover a device-disjoint test set, which is why Section 5.3 treats missing provenance as a first-class dataset defect.

import pandas as pd
df.to_parquet("telemetry.parquet", compression="zstd")   # schema, stats,
                                                          # chunking, all handled
cols = pd.read_parquet("telemetry.parquet", columns=["rpm", "oil_temp"])

What's Next

In Section 5.2, we take these correctly-stored streams and carve them into the windows a model actually trains on: how long a window should be, how much they overlap, and how a label gets attached to a slice of time without smearing across boundaries. The self-describing descriptors you built here are exactly what make that windowing reproducible instead of a pile of magic numbers.