Missingness-as-signal and personalized baselines

"The user has not worn the ring for three nights. I have logged this as missing data. My physician colleague logged it as the flu."

An Over-Literal Sleep Tracker

Section 33.1 established that longitudinal wearable data is irregular and sparse, and showed how to represent an unevenly-sampled stream. Here we ask the two questions that irregular sampling forces on you but does not answer: what does a gap mean, and what should we compare a present value against? The probabilistic vocabulary comes from Chapter 4, and the change-detection instinct from Chapter 12; this section specializes both to the peculiar economics of a body that generates its own noise and wears its own sensor on its own schedule.

Missingness is rarely random

What the three regimes are. Statisticians sort missingness into three mechanisms, and the distinction is not academic here because it decides whether you may safely ignore a gap. Data is missing completely at random (MCAR) when the gap is independent of everything (a cosmic-ray bit flip). It is missing at random (MAR) when the gap depends only on observed variables (a sensor that drops out above a known temperature you also record). It is missing not at random (MNAR) when the probability of the gap depends on the unobserved value itself or on the hidden state you care about. Wearable health data is overwhelmingly MNAR: people stop wearing devices precisely when they are sick, hospitalized, depressed, or traveling, and optical contact fails precisely during the vigorous motion or vasoconstriction that also shifts the vital sign.

Why MNAR forbids naive imputation. The standard reflex (fill the hole with the last value, the mean, or a spline) silently assumes MCAR or MAR. Under MNAR that reflex destroys the signal: imputing a "normal" resting heart rate across the three nights a user stopped wearing their ring because they had influenza erases the very episode a health model exists to catch. The correct move is to stop treating the gap as an absence and start treating its pattern (its timing, duration, and co-occurrence) as an observed feature in its own right. Non-wear time, charging gaps, and quality-gated dropouts are data.

Encoding the gap: masks, deltas, and decay

The core mechanism. The cleanest way to feed missingness to a sequence model is the GRU-D decay trick: rather than carrying a stale observation forward unchanged, decay it toward the person's running mean at a rate the network learns per feature. Concretely, define an exponential decay weight from the elapsed time,

where \(x_{t'}\) is the last observed value, \(\bar{x}\) is the empirical (ideally per-person) mean, \(\delta_t\) is the time since that last observation, and \(w,b\) are learned. When a value is fresh (\(\delta_t\) small) \(\gamma_t \to 1\) and the model trusts the carried value; when the gap is long \(\gamma_t \to 0\) and the estimate relaxes to the baseline while the mask still tells the model this was a guess. This gives the recurrent and temporal-convolutional models of Chapter 14 a principled, differentiable way to consume irregular health streams without pretending the holes are not there.

import numpy as np

# One user's hourly resting-HR stream; NaN = not worn / quality-gated out.
x = np.array([58, 57, np.nan, np.nan, np.nan, 61, 63, np.nan, 72, 74, 73])
mask  = (~np.isnan(x)).astype(float)                 # 1 = observed, 0 = gap
# time since last observation (delta), in samples
delta = np.zeros_like(x)
for k in range(1, len(x)):
    delta[k] = 0 if mask[k] else delta[k-1] + 1

def gru_d_impute(x, mask, delta, x_bar, w=0.5, b=0.0):
    x_hat, last = np.empty_like(x), x_bar
    for k in range(len(x)):
        if mask[k]:                                  # observed: use it, refresh memory
            x_hat[k] = x[k]; last = x[k]
        else:                                        # gap: decay last toward baseline
            gamma = np.exp(-max(0.0, w*delta[k] + b))
            x_hat[k] = gamma*last + (1-gamma)*x_bar
    return x_hat

x_bar = np.nanmean(x[:6])                             # personal early-window mean
print(np.round(gru_d_impute(x, mask, delta, x_bar), 1))
print("mask :", mask.astype(int))
print("delta:", delta.astype(int))

The printout shows the decay in action: the three-hour gap after the 57 does not freeze at 57, it drifts toward the person's mean, and the mask and delta rows remain available so the model never confuses a decayed guess with a real measurement. Crucially, we never delete the fact that a gap happened, we feature-ize it.

from pypots.imputation import SAITS
model = SAITS(n_steps=48, n_features=6, n_layers=2, d_model=128, epochs=10)
model.fit({"X": X_with_nans})                 # NaNs are the mask; deltas handled internally
X_imputed = model.impute({"X": X_with_nans})  # missingness-aware, not mean-fill

Personalized baselines: the individual is the yardstick

Why population norms fail. Between-person physiological variation dwarfs the within-person deviations that signal a health event. A resting heart rate of 50 is an alarm for a sedentary 70-year-old and a resting norm for a cyclist; a nocturnal skin temperature of 36.2 degrees Celsius is a person's baseline or a half-degree elevation depending entirely on their own history. A model that flags departures from a population mean will drown in false positives for the tails of the distribution and miss real events for people whose baseline sits far from the crowd. The fix is to make each person their own control.

How to build one. The workhorse is a per-person robust standardization: maintain a rolling estimate of the individual's location and scale, then score new samples as deviations from their center. Using the median and the median absolute deviation (MAD) rather than mean and standard deviation keeps the baseline from being dragged by the very anomalies you want to detect,

where \(\tilde{\mu}^{(u)}_t\) is user \(u\)'s rolling median and the constant rescales MAD to match a standard deviation under normality. A large \(|z_t^{(u)}|\) means "unusual for this person," which is what an elevated resting heart rate over a personal baseline (a widely used early-illness signal) actually is. The change-point and control-chart tools of Chapter 12 then operate on \(z_t^{(u)}\) instead of the raw stream.

The cold-start problem, and why hierarchy solves it. A brand-new user has no personal history, so their baseline is undefined for days or weeks. The principled remedy is a hierarchical (partial-pooling) estimator: start each user's baseline at the population prior and shrink toward their own accumulating data as evidence arrives, \(\hat{\mu}^{(u)} = \lambda_u\,\bar{x}^{(u)} + (1-\lambda_u)\,\mu_{\text{pop}}\), with the personal weight \(\lambda_u\) growing as the user logs more valid samples. Early on you borrow strength from the population; later the person speaks for themselves. This is the empirical-Bayes shrinkage of Chapter 4 applied to onboarding, and it is why good wearables show conservative insights in week one and sharper ones by month two.

What's Next

In Section 33.3, we stop treating the wearable as a single gappy channel and start fusing several at once: PPG, accelerometer, skin temperature, and electrodermal activity together. Multimodal fusion both sharpens the personal baseline (motion context explains away a heart-rate spike) and complicates the missingness picture (now different modalities drop out at different times), setting up the missing-modality robustness that becomes a central theme later in the book.