Reproducibility and field-validation protocols

"It worked on my laptop, on a Tuesday, with a random seed I have since misplaced."

A Reproducible AI Agent

Three R's, and why sensing makes them harder

The literature separates three distinct claims that casual usage conflates. Repeatability: the same team, same code, same data, same machine gets the same number. Reproducibility: a different team, using your released code and data, reaches the same conclusion. Replicability: an independent team, collecting fresh data and writing fresh code, reaches the same conclusion. They form a ladder of increasing strength, and a benchmark that clears only the first rung is fragile. What/why: you separate them because each requires a different artifact. Repeatability needs pinned randomness and environments; reproducibility needs released, versioned data and code; replicability needs a clearly specified protocol, not just an implementation.

Sensing adds hazards that pure software or vision benchmarks rarely face. Firmware revisions silently change a sensor's filtering or gain; a clock drift of a few milliseconds reorders events and quietly breaks time-aware folds (Section 65.3); a driver update alters resampling; hardware ages between the day you trained and the day you audit. Nondeterministic GPU kernels, unpinned library versions, and preprocessing that depends on floating-point reduction order all inject variance that has nothing to do with the science. Bouthillier and colleagues showed that this uncontrolled variance is large enough to flip benchmark rankings, so a single-seed comparison is not evidence of anything. The remedy is to treat every source of variance as a named, logged, controlled variable.

The reproducibility toolkit

Concretely, a reproducible sensing evaluation captures five things and stamps them onto every reported number. (1) Seeds for every random source (Python, NumPy, the framework, and the data loader workers), plus a determinism flag where the framework offers one; note that full determinism can cost throughput, so you decide it per run rather than globally. (2) Environment: exact library versions, CUDA and driver build, OS, and hardware, ideally frozen in a container image so the audit runs the same stack. (3) Data version: a content hash of the exact split files, not just a dataset name, because "MHEALTH" or "PTB-XL" says nothing about which subjects landed in which fold. (4) Code version: the commit hash, with a clean or explicitly-recorded working tree. (5) Config: the full hyperparameter and preprocessing configuration as data, never as edited-in-place constants.

Why insist on hashing the split rather than naming the dataset? Because the entire chapter has argued that how you split (by user, site, device, time) determines the number more than the model does. A reproducibility manifest that omits the split is reproducing the wrong thing. The function below computes a single run fingerprint from these ingredients; if two runs disagree on the number but share a fingerprint, you have found nondeterminism, and if they share the number but not the fingerprint, you have found a coincidence you should not trust.

import hashlib, json, platform, subprocess

def run_fingerprint(split_files, config, seed):
    """A content-addressed fingerprint of everything that fixes a result."""
    h = hashlib.sha256()
    for path in sorted(split_files):            # hash the EXACT split, not its name
        with open(path, "rb") as f:
            h.update(hashlib.sha256(f.read()).digest())
    env = {
        "python": platform.python_version(),
        "platform": platform.platform(),
        "commit": subprocess.run(["git", "rev-parse", "HEAD"],
                                 capture_output=True, text=True).stdout.strip(),
        "seed": seed,
        "config": config,                        # full hyperparams + preprocessing
    }
    h.update(json.dumps(env, sort_keys=True).encode())
    return h.hexdigest()[:16], env

fp, env = run_fingerprint(["folds/test_site_A.parquet"], {"lr": 3e-4, "win": 256}, seed=0)
print("run fingerprint:", fp)

The fingerprint above is the honest minimum, and it makes the mechanism visible, but you would not maintain it by hand across hundreds of runs.

Pre-registration, metric suites, and reporting standards

Reproducing a number is worthless if the number was chosen after seeing the results. The subtle leak here is not in the data but in the analysis: trying twenty splits, five metrics, and three thresholds, then reporting the flattering combination. The defense is pre-registration: before you touch the test set, write down the split protocol, the primary metric, the subgroup breakdowns, and the decision rule that says "ship" or "do not ship." The test set is unsealed once. Kapoor and Narayanan document how the absence of this discipline produced a wave of ML-based science that failed to replicate, almost always through some form of leakage or post-hoc metric selection, which is exactly the failure mode this whole chapter is built to prevent (Chapter 5).

Alongside the metric plan, ship the two standard reporting artifacts. A datasheet for the dataset records provenance, sensor models, sampling rates, demographics, and known biases; a model card records intended use, the exact evaluation protocol, per-subgroup performance, and out-of-scope conditions. For sensing these are not bureaucratic: "works on wrist-worn PPG, indoors, resting heart rate 50 to 100 bpm, not validated during exercise" is the difference between a safe claim and a recall. Report variance too, never a single number: run multiple seeds and, where feasible, resample the subjects, then quote a mean with a confidence interval bootstrapped over subjects rather than windows.

From benchmark to field: prospective validation

A leakage-safe benchmark is a necessary rehearsal, not the performance. Field validation is the prospective study that earns deployment trust, and it follows a staged protocol. First, shadow mode: run the model live on real device streams but suppress its actions, comparing predictions against outcomes without any risk. Next, reference-standard agreement: for a subset of field data, obtain trustworthy labels from an instrument you trust more (a clinical 12-lead ECG behind a wrist PPG, a survey-grade GNSS behind a consumer tracker, a load cell behind an inferred force). Then a staged rollout to a small, monitored cohort with pre-set stopping rules, widening only as the field metrics hold. Throughout, watch the operational metrics of Section 65.1 and the drift signals that feed Chapter 66, because the field is where covariate shift first shows up.

For continuous measurements, agreement is quantified with a Bland-Altman analysis, not a correlation. Correlation can be high while a device reads systematically 8 bpm low; Bland-Altman instead plots the difference between the device and the reference against their mean, then reports the bias \(\bar{d}\) and the 95% limits of agreement \(\bar{d} \pm 1.96\,s_d\). Those limits, not \(R^2\), tell a clinician whether the error is tolerable for the decision at hand. When ground truth cannot be collected live, a validated simulator or digital twin (Chapter 55) can pre-screen failure modes, though it never replaces a prospective field study for a regulated claim.

What's Next

In Chapter 66, we confront the reason a reproducible, field-validated model still degrades: the world moves. We formalize covariate, label, and concept shift for sensor streams, detect when inputs fall out of distribution, and adapt models at test time (TENT, CoTTA) while monitoring the risk that adaptation itself can introduce.