"A result you cannot reproduce is a rumor with a confidence interval."
A Reproducible AI Agent
The Big Picture
The previous six sections taught you to choose formats, cut windows, avoid leakage, build honest splits, calibrate per device, and reason about label quality. Each of those is a decision, and every decision is a place where a future teammate (or future you) can silently do something different and get a different number. This section is about freezing those decisions so the same raw bytes always produce the same dataset, and so a downstream consumer can detect the day your sensors start lying. Two ideas carry the weight: a versioned pipeline that makes the transformation from raw stream to training tensor deterministic and auditable, and a data contract that states, in machine-checkable form, what each stage promises about the data it emits. Together they turn "it worked on my laptop in March" into a build you can rerun in one command a year later.
This section assumes you are comfortable with the sensor formats and timestamping of Chapter 3 and with the windowing, split, and calibration choices from Sections 5.2 through 5.6. We treat that entire chain as the thing to be versioned. We stay out of the way of Chapter 69 (operating deployed models across a fleet) and Chapter 65 (evaluation protocols); here the deliverable is a dataset build that anyone can regenerate byte-for-byte and trust on arrival.
Why sensor pipelines rot, and what reproducibility actually requires
What. A sensor pipeline is the ordered sequence of transformations that turns raw device output into model-ready tensors: decode, resample, synchronize, filter, window, normalize, label, split. Why it rots. Every stage hides a moving part. A resampler pulls in a new SciPy release with a different default; a firmware update changes the accelerometer full-scale range; someone re-runs the split with a fresh random seed; a timezone assumption flips at a daylight-saving boundary. None of these throw an error. They just quietly change the dataset, and six weeks later a model that "regressed" is really a model trained on different data.
How reproducibility is achieved. Determinism is not one thing but four, and you need all four to actually hold. First, fix the inputs: content-address the raw data so a hash identifies the exact bytes, not a mutable path. Second, fix the code: pin the pipeline to a commit and pin every library version. Third, fix the configuration: put window length, stride, seed, filter cutoffs, and split policy in a versioned config file, never in a notebook cell. Fourth, fix the environment and randomness: seed every random operation and record the interpreter and OS. The output identity you want is a function of all four:
$$\text{dataset\_id} = H\big(\text{raw\_hash},\ \text{code\_commit},\ \text{config\_hash},\ \text{env\_hash}\big)$$where \(H\) is a cryptographic hash. If any input changes, the id changes, and you have caught a silent mutation before it reached a model. If all four match, re-running the build is guaranteed to reproduce the same tensors, which is the whole game.
Key Insight
The reproducibility failure that actually bites sensor teams is almost never model randomness; it is silent input drift. A model retrain is loud and expected. A resampler that changed its interpolation default, or a raw-data directory someone quietly re-uploaded with a bug fix, changes your data with no signal at all. Content-addressing the raw bytes and hashing the config is worth more than a hundred fixed seeds, because it converts an undetectable change into a changed id that fails a check. Reproducibility is less about repeating success and more about making unrequested change impossible to hide.
Data contracts: promises the pipeline must keep
What. A data contract is an explicit, machine-checkable specification of what a dataset or a pipeline stage guarantees: the schema (channel names, dtypes, units), the physical ranges, the sampling rate and its allowed jitter, the null and gap policy, and the label vocabulary. It is the API of your data. Why. Without a contract, every consumer re-discovers the data's quirks by trial and error, and a producer can break a downstream model just by shipping a new sensor batch in millivolts instead of volts. A contract moves that failure to the boundary where it belongs: the moment bad data arrives, a validation gate rejects it with a specific reason, instead of a model degrading mysteriously a month later.
How. A contract has two halves. The declarative schema lists fields, types, and units and is checked cheaply on every batch. The statistical expectations assert distributional facts: the accelerometer magnitude at rest sits near \(9.81\ \mathrm{m/s^2}\), the fraction of missing samples per window stays below a threshold, the inter-sample interval clusters at \(\Delta t = 1/f_s\). These expectations connect directly to the sensor physics of Chapter 2: a contract that knows gravity's magnitude can flag a mislabeled axis or a wrong full-scale setting automatically. When. Enforce the contract at three gates: at ingestion (raw data crossing into your system), between stages (each transform validates its output), and at the training boundary (the final tensor matches what the model expects). Contracts should fail closed: a violation halts the build rather than passing suspect data through.
Practical Example: the wind farm that shipped a unit change
An industrial team monitoring gearbox vibration on a fleet of wind turbines had a stable anomaly detector running for a year. One maintenance cycle, a vendor replaced a batch of accelerometers with a newer model that reported acceleration in \(g\) rather than \(\mathrm{m/s^2}\), a factor of about \(9.81\). No code changed, no error fired, and the ingestion path happily accepted the new files. The detector's false-alarm rate collapsed toward zero because every reading now looked nine times quieter than the trained baseline, so real faults slid under the threshold. A one-line data contract, "root-mean-square vibration on a healthy unit lies in \([2, 40]\ \mathrm{m/s^2}\)", would have rejected the very first file from the new sensors and named the reason. After the incident the team added unit and range checks at ingestion; the next hardware swap was caught in minutes, not quarters. This is the same class of failure the normalization discussion in Chapter 37 revisits under real drift.
Building a versioned pipeline in practice
How to structure it. Express the pipeline as a directed acyclic graph of stages, each a pure function from an input artifact to an output artifact, each tagged with the hash of its inputs plus its config. This buys you two things automatically: caching (a stage whose input hash is unchanged is skipped, so rebuilds are cheap) and lineage (every tensor can name the raw file, code commit, and config that produced it). When a reviewer asks "where did this training example come from", the answer is a lookup, not an archaeology project. When to version what. Version the raw data (immutably, content-addressed), the code (git commit), the config (in the repo), and the resulting dataset (a manifest recording the four-part id above). Do not version derived tensors by copying them everywhere; version the recipe that regenerates them and cache the output.
The snippet below computes the dataset identity from its four inputs and writes a manifest. It is deliberately small; the point is that the identity is a deterministic function of pinned inputs, so two people running it on the same commit and config get the same id and can prove they built the same data.
import hashlib, json, subprocess, sys, platform
from pathlib import Path
def sha256_file(path, chunk=1 << 20):
h = hashlib.sha256()
with open(path, "rb") as f:
for block in iter(lambda: f.read(chunk), b""):
h.update(block)
return h.hexdigest()
def hash_str(s):
return hashlib.sha256(s.encode()).hexdigest()
def build_manifest(raw_files, config):
raw_hash = hash_str("".join(sorted(sha256_file(p) for p in raw_files)))
code_commit = subprocess.check_output(
["git", "rev-parse", "HEAD"]).decode().strip()
config_hash = hash_str(json.dumps(config, sort_keys=True))
env_hash = hash_str(f"{sys.version}|{platform.platform()}")
dataset_id = hash_str(raw_hash + code_commit + config_hash + env_hash)[:16]
return {
"dataset_id": dataset_id,
"raw_hash": raw_hash, "code_commit": code_commit,
"config_hash": config_hash, "env_hash": env_hash,
"config": config,
}
if __name__ == "__main__":
cfg = {"window_len": 256, "stride": 64, "fs_hz": 50,
"split": "subject_disjoint", "seed": 7}
manifest = build_manifest(sorted(Path("raw").glob("*.parquet")), cfg)
Path("dataset_manifest.json").write_text(json.dumps(manifest, indent=2))
print("dataset_id:", manifest["dataset_id"])
dataset_id from raw-data hashes, the git commit, a sorted config hash, and an environment fingerprint. Re-running on the same inputs reproduces the id; any silent change to raw bytes, code, config, or interpreter flips it, which is exactly the alarm you want.Note the two subtle correctness moves in the code. Raw hashes are sorted before concatenation so file order does not affect the id, and the config is serialized with sort_keys=True so a reordered dictionary still hashes identically. Both prevent spurious id changes that would train teams to ignore the alarm. This same discipline underpins the leakage-safe benchmarking of Chapter 65: a benchmark is only fair if everyone provably built the same splits.
Right Tool: let a data-versioning stack carry the plumbing
The manifest above is instructive but it is roughly 30 lines you would then have to grow into caching, remote storage, and a lineage graph, easily a few hundred more. Purpose-built tools do this for you. DVC turns dvc add raw/ plus a short dvc.yaml stage file into content-addressed raw data, cached deterministic stages, and a git-tracked lineage graph in under ten lines of config; a rebuild after an unchanged input is a no-op it detects automatically. For the contract half, Pandera or Great Expectations expresses a full schema-plus-range check as a short declarative object instead of dozens of hand-written asserts, and produces a readable failure report naming the offending column and rows. The rule of thumb: hand-roll the manifest once so you understand the four-part id, then adopt the tools so caching, storage, and validation reports are not your code to maintain.
When reproducibility is worth the cost, and when it is not
What it costs. Full versioning adds friction: hashing large raw archives takes time, pinned environments must be rebuilt, and a contract that fails closed will occasionally block a build at an inconvenient hour. When to pay it. Pay in full whenever a result will be published, audited, or shipped to a device you cannot easily update, and whenever more than one person touches the pipeline. A clinical study, a regulated wearable (the validation regime of Chapter 34), and any leaderboard submission all demand it, because the question "is this the same data" must have a provable answer. When to lighten up. A solo exploratory notebook that will be thrown away next week does not need a content-addressed remote; pin the seed and the library versions and move on. The skill is matching the ceremony to the stakes: enough determinism that you can trust and defend the number, not so much that exploration grinds to a halt.
Exercise
Take a small human-activity dataset and build it twice through the same windowing and split code, but on the second run bump one library (say, change your resampling library's minor version) without touching your own code. Compute the dataset_id both times using the manifest approach above. Confirm the id changes even though your code did not, then write the two-line data contract (sampling interval and accelerometer-magnitude-at-rest range) that would have flagged whichever run produced physically wrong values. Report which of the four id components differed and why.
Self-Check
1. A colleague says "I fixed all the random seeds, so my dataset is reproducible." Name one common way the dataset can still silently change, and how content-addressing would catch it.
2. Why should a data contract be enforced at ingestion rather than only at the training boundary? Give a failure that only an ingestion-time check catches early.
3. In the dataset_id formula, why are the raw file hashes sorted and the config serialized with sorted keys before hashing? What false alarm does each choice prevent?
Lab 5
build leakage-safe subject- and device-disjoint splits for human activity recognition; measure the accuracy gap vs a naive random split.
Bibliography
Reproducibility and pipeline versioning
Kuprieiev, R. et al. (2024). DVC: Data Version Control. Documentation and open-source project.
The canonical tool for content-addressed data, cached deterministic pipeline stages, and git-tracked lineage; the practical realization of the four-part dataset id in this section.
A case-study survey of what makes computational workflows reproducible in practice; grounds the code, config, environment, and data pinning that this section formalizes.
Data contracts and validation
Great Expectations (2024). An open-source framework for data validation and documentation.
Expresses schema plus statistical expectations as declarative checks with human-readable failure reports; the tooling behind the fail-closed contract gates described here.
A lightweight schema-and-range validation library for tabular sensor data; a few lines replace dozens of hand-written asserts at each stage boundary.
Google's production data-validation system for ML; formalizes schema inference, anomaly detection, and the training-serving skew that data contracts are designed to prevent.
Experiment tracking and ML operations
Introduces run tracking, parameters, and artifact logging that pair with a versioned pipeline to make a training result auditable end to end.
The classic account of how unstable data dependencies and undeclared consumers accumulate silent risk; the intellectual case for data contracts and versioned inputs.
Gebru, T. et al. (2021). Datasheets for Datasets. Communications of the ACM.
Argues that every dataset should ship documented provenance, composition, and intended use; the human-readable companion to the machine-checkable contract.
What's Next
In Chapter 6, we open Part II and start turning clean, versioned streams into clean signals: moving averages, FIR and IIR filters, and the band-pass and notch designs that strip noise while preserving the phenomenon. Every filter you meet there is a pipeline stage in the sense of this section, with a cutoff and a design choice that belongs in your versioned config and under a data contract; reproducibility is not left behind when we move from data engineering to signal processing, it comes along as a habit.