Model versioning and deployment

"You asked which version of me is running. I am four versions, one per device class, and two of them think the accelerometer is in millig."

An Incompletely Rolled-Out AI Agent

This section assumes you have read the sensor data contract in Section 69.1, which fixes the schema, units, and ranges a stream must satisfy, and that you build models on leakage-safe datasets from Chapter 5. The versioning discipline below is the operational counterpart of that contract: the data contract governs what flows in, and the model version governs what decides on it. Get monitoring for what happens next in Section 69.3.

What must be versioned together: the deployable unit

The core mistake is versioning the weights alone. A sensor model's behavior is determined by an entire pipeline, and any component that changes the mapping from raw counts to decision is part of the version. The deployable unit for a sensor model is the tuple of: the trained weights; the exact preprocessing graph (windowing, resampling, filter coefficients, normalization statistics); the calibration parameters that turn ADC counts into physical units; the input contract it expects (schema, units, sample rate); the target hardware and runtime (the quantized graph for that accelerator, from Chapter 59); and the evaluation report that certified it. If any element changes, the behavior can change, so the version must change.

Concretely, adopt semantic versioning with sensor semantics. A major bump means the input contract changed and old data is incompatible (you moved from 50 Hz to 100 Hz input, or swapped the feature set). A minor bump means retrained weights on a compatible contract (new training data, same interface). A patch means a hardware-specific rebuild with identical float behavior (re-quantized for a new chip). This lets a fleet operator answer the question that actually matters during an incident: "can device A's stream be fed to the model that device B is running?" The answer is yes only when the major version and contract match.

The registry, lineage, and reproducible identity

A model registry is the fleet's source of truth: an append-only record mapping each version string to its artifacts, its training run, and its promotion state (staging, canary, production, retired). What makes it trustworthy is lineage: every version links back to the exact dataset snapshot, the git commit of the training code, the hyperparameters, and the leakage-safe split protocol used to evaluate it. When a regulator or a post-incident review asks "how was the model on this device trained and tested," lineage is the answer, and for safety-relevant perception it is a requirement, not a nicety (see the safety-case discussion in Chapter 68).

The practical anchor is a content-addressed manifest: hash every artifact, hash the manifest, and let that digest be the model's true identity. Two builds with the same digest are provably the same decision function; two builds that differ anywhere produce different digests and cannot masquerade as each other. The code below builds such a manifest and shows how a device verifies compatibility before it ever runs an inference.

import hashlib, json
from dataclasses import dataclass, asdict

def sha256_file(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 16), b""):
            h.update(chunk)
    return h.hexdigest()

@dataclass(frozen=True)
class ModelManifest:
    version: str                 # semantic: MAJOR.MINOR.PATCH
    contract_id: str             # must match the data contract from 69.1
    input_units: dict            # {"accel": "m/s^2", "gyro": "rad/s"}
    sample_rate_hz: int
    target_runtime: str          # e.g. "tflite-int8/hexagon-v66"
    weights_sha: str
    preprocessing_sha: str       # windowing + filter coeffs + norm stats
    calibration_sha: str         # per-unit or per-model calibration
    eval_report_sha: str         # leakage-safe evaluation that certified it

    def digest(self):
        payload = json.dumps(asdict(self), sort_keys=True).encode()
        return hashlib.sha256(payload).hexdigest()[:16]

def device_can_run(manifest, device):
    """A device runs a model only if contract, units, rate, and runtime all agree."""
    reasons = []
    if manifest.contract_id != device["contract_id"]:
        reasons.append("contract mismatch (major-version incompatible)")
    if manifest.input_units != device["input_units"]:
        reasons.append(f"unit mismatch: model wants {manifest.input_units}, "
                        f"device emits {device['input_units']}")
    if manifest.sample_rate_hz != device["sample_rate_hz"]:
        reasons.append("sample-rate mismatch")
    if manifest.target_runtime != device["runtime"]:
        reasons.append("runtime/accelerator mismatch")
    return (len(reasons) == 0, reasons)

m = ModelManifest(
    version="2.3.1", contract_id="imu-wrist-v2",
    input_units={"accel": "m/s^2", "gyro": "rad/s"}, sample_rate_hz=50,
    target_runtime="tflite-int8/hexagon-v66",
    weights_sha="…", preprocessing_sha="…",
    calibration_sha="…", eval_report_sha="…")

device = {"contract_id": "imu-wrist-v2",
          "input_units": {"accel": "millig", "gyro": "rad/s"},  # firmware bug!
          "sample_rate_hz": 50, "runtime": "tflite-int8/hexagon-v66"}

ok, reasons = device_can_run(m, device)
print(m.digest(), ok, reasons)
# -> a stable 16-char id, False, ['unit mismatch: model wants ... device emits millig ...']

The manifest above is the object the registry stores and the device verifies. Notice that the compatibility gate catches the epigraph's failure: a device whose firmware reports millig refuses to load a model that expects m/s^2, rather than running and producing confidently wrong output. This is the whole point of tying the model version to the data contract of Section 69.1.

Deploying to a heterogeneous fleet: canary, shadow, and staged rollout

A sensor fleet is not one server; it is thousands of devices spanning firmware revisions, hardware generations, and physical environments. Pushing a new version to all of them at once is the single riskiest action in fleet operations, because a regression can brick perception on every unit before the first alert fires. Three patterns tame this. In shadow mode, the new version runs alongside the incumbent and its outputs are logged but not acted on, so you compare decisions on live data with zero user risk. In a canary, you route a small, representative slice of the fleet (say one percent, stratified across device generations and sites) to the new version and hold it there while health metrics accumulate. In a staged rollout, you widen exposure in waves, gating each wave on the previous wave's metrics, with an automatic rollback to the pinned previous version if a guardrail trips.

The stratification is the sensor-specific twist. A canary that samples one percent of devices uniformly can entirely miss the one hardware generation where the new quantization breaks, because that generation is three percent of the fleet. Stratify the canary by the axes that carry sensor nuisance (device model, firmware build, site, and mounting), the same identities that structured your evaluation splits in Chapter 65. Because true labels arrive late or never in the field, canary health leans on the proxy signals developed in Section 69.4: prediction-distribution shift, confidence collapse, disagreement with the shadowed incumbent, and downstream action rates.

Rollback, pinning, and the boring guarantees

The deployment mechanism is only as good as its reverse gear. Two guarantees make a fleet operable. First, every device can be pinned to a specific, previously certified version and will refuse to auto-update past it, so a known-good state is always reachable in one command. Second, rollback is atomic and version-complete: reverting the weights also reverts the preprocessing, calibration, and contract expectations as one unit, because a half-rolled-back model (new weights, old normalization) is a configuration nobody tested. This is why the deployable unit must be atomic from the start; you cannot roll back what you versioned separately. Keep at least the last known-good version resident on the device so rollback does not depend on connectivity, which for intermittently connected sensors (Chapter 63) can be days away.

What's Next

In Section 69.3, we watch the version we just deployed while it runs: how to detect input drift and data-quality decay across a fleet, so that a model still nominally "in production" is caught the moment the physical world stops matching the data it was certified on.