Normalization and per-device calibration

"Two sensors reading the same world will still disagree. Normalization is how you argue them into a common language without whispering the test answers on the way."

A Diplomatic AI Agent

This section assumes you understand sensor bias, gain, and drift as physical quantities, developed in Chapter 2, and the leakage-safe split boundaries (device, user, site) from Section 5.4. Here we take windowed, split data and ask how to scale and calibrate it so that a model trained on one set of devices behaves on another. We deliberately treat model-output calibration (making predicted probabilities match observed frequencies) as a separate topic, covered in Chapter 18; the word "calibration" here means the hardware sense.

Why raw units mislead a model

What. Most sensor channels arrive in physical or raw-count units that mix three things: the signal, a per-device offset, and a per-device scale. A single reading can be modeled as \(x = g\,(s + b) + \varepsilon\), where \(s\) is the true stimulus, \(b\) is a bias, \(g\) is a gain, and \(\varepsilon\) is noise. Why it matters. Gradient-based models converge poorly when channels span wildly different magnitudes: an accelerometer in \(\pm 20\ \text{m/s}^2\) and a barometer in \(\sim 10^5\ \text{Pa}\) fed into the same network will let the barometer dominate every dot product until the optimizer painfully rescales its own weights. Worse, if \(b\) and \(g\) differ per device, a model that learned "class A sits near value 3.1" on the training devices will be wrong on a test device whose baseline sits at 3.4. How. Normalization fixes the magnitude problem; calibration fixes the per-device offset-and-gain problem. They are related but not the same operation, and conflating them is where subtle bugs live.

Normalization schemes and where the statistics come from

What. The workhorses are per-channel z-score (subtract mean, divide by standard deviation), min-max (rescale to \([0,1]\)), and robust scaling (subtract median, divide by interquartile range). For a channel \(c\), z-scoring is

where \(\mu_c\) and \(\sigma_c\) are estimated statistics. When to prefer which. Use z-score as the default for roughly symmetric signals; use robust scaling when outliers and spikes are common (vibration, EMG bursts), because a single clipping event can wreck a standard deviation; use min-max only when a signal has hard physical bounds and you truly want to preserve them. How, and the part everyone gets wrong. The statistics \(\mu_c, \sigma_c\) are learned parameters of your pipeline, so they must be estimated on the training split alone and then applied unchanged to validation and test. Estimating them over the full dataset lets the test set's mean and variance leak backward into training, which is exactly the failure Section 5.3 warns against.

Per-device calibration: making devices speak the same units

What. Calibration removes the sensor-specific transfer function so that a known input maps to a known output on every unit. The classic two-point linear calibration solves for gain and offset from two reference stimuli \(s_1, s_2\) with measured responses \(x_1, x_2\): \(g = (x_2 - x_1)/(s_2 - s_1)\) and \(b = x_1 - g\,s_1\), then recovers \(\hat{s} = (x - b)/g\). Why it is separate from normalization. Normalization uses statistics of your data; calibration uses references from the physical world (a level surface for an IMU, ice and boiling water for a thermistor, a zero-gas purge for a chemical sensor). Calibration can be done per device with no labels at all, which is powerful, because it attacks device shift at its source rather than hoping the model averages it out. When you lack references. If you cannot bench-calibrate, per-device normalization is the pragmatic stand-in: compute each device's own baseline statistics (from a resting period, or a rolling window) and normalize each device by its own numbers. This does not need the true \(g, b\), only that each device is made internally consistent.

Instance normalization and the shift it defends against

What. Instead of (or in addition to) a fixed dataset scaler, you can normalize each window by its own statistics at inference time: subtract the window mean and divide by its standard deviation per channel. This is instance normalization, and its reversible form (normalize the input, then add the statistics back to the output) is known in the forecasting literature as reversible instance normalization. Why. It removes slow per-window offset and scale shifts that a global scaler cannot track, which is exactly the drift that breaks cross-device and cross-day generalization. When. Prefer instance normalization when the absolute level carries little class information and the shape does the work (activity recognition, many fault-detection tasks); avoid it when the absolute level is the signal (a barometer for altitude, a thermometer for fever), because you would normalize away the very thing you want to predict. This is the same augmentation-and-normalization design space that Chapter 13 revisits for deep models.

The code below shows the leakage-safe pattern end to end: fit a per-channel scaler on the training devices only, then apply a per-device baseline correction that each device computes from its own resting statistics. Note that nothing about the validation devices touches the fitted scaler's parameters.

import numpy as np

class LeakageSafeScaler:
    """Per-channel z-score fit on TRAIN groups only, plus per-device debias."""
    def fit(self, X_train):                    # X_train: (N, L, C) train windows
        flat = X_train.reshape(-1, X_train.shape[-1])
        self.mu = flat.mean(0)                  # (C,) learned on train ONLY
        self.sigma = flat.std(0) + 1e-8
        return self

    def transform(self, X, device_baseline=None):
        Xc = X.astype(np.float64)
        if device_baseline is not None:         # per-device offset removal
            Xc = Xc - device_baseline           # (C,) this device's own zero
        return (Xc - self.mu) / self.sigma

# Fit on training devices, freeze, then transform each device by ITS baseline.
scaler = LeakageSafeScaler().fit(X_train)       # never sees val/test devices
base_val = X_val_rest.mean((0, 1))              # this device's resting mean (C,)
X_val_norm = scaler.transform(X_val, device_baseline=base_val)
print(X_train.shape[-1], "channels; scaler frozen on train")

Normalization and calibration are the last preprocessing step before modeling and the one most likely to quietly corrupt an otherwise careful pipeline. Treat the scaler as a fitted model with a strict train-only fitting rule, keep hardware calibration distinct from data-statistics normalization, and push per-device corrections down to the device when its bias is private and time-varying. These habits are what let a model cross the gap from the devices it trained on to the ones it will actually run on.

What's Next

In Section 5.6, we turn from the signal side to the label side: annotation quality, weak and noisy labels, and the label delay that shifts your ground truth in time. We will see why a perfectly normalized dataset with sloppy labels still trains a sloppy model, and how to reason about label noise as its own source of error.