Clocks, timestamps, and multi-sensor synchronization

"Each of my sensors swears it knows the time. The trouble is that no two of them agree, and none of them will admit it."

A Multi-Sensor AI Agent

Section 3.1 treated a single stream sampled on a single clock, and Section 3.2 contrasted clock-driven with event-driven acquisition. Here we assume the readings exist and ask the question that fusion forces on us: how do we place samples from independent devices on a common axis? This presumes a little probability vocabulary for talking about timing error, which the Chapter 4 uncertainty primer supplies, and it feeds directly into Chapter 48, where misalignment becomes fusion error you can measure.

What a clock actually is: offset, skew, and drift

A sensor's clock is a counter driven by an oscillator, usually a quartz crystal, that ticks at a nominal frequency. It is never exactly right. We model device \(i\)'s reported time \(C_i\) as an affine function of true time \(t\):

Here \(\theta_i\) is the offset (how far the clock is ahead or behind at \(t=0\)) and \(\epsilon_i\) is the skew (its fractional rate error, the reason it gains or loses time). Skew is quoted in parts per million (ppm): a garden-variety consumer crystal is rated at \(\pm 20\) to \(\pm 50\) ppm. That sounds negligible until you unpack the unit. One ppm is one microsecond of error per second, which is about \(86\ \mathrm{ms}\) per day. A 50 ppm crystal can therefore drift more than four seconds in a single day if left uncorrected. Two independent 50 ppm clocks can move apart at up to 100 ppm relative to each other. Drift is the slow change of \(\epsilon_i\) itself, driven mostly by temperature: the same crystal that reads 20 ppm on a bench can swing tens of ppm as a device warms in a pocket or a bearing housing.

The practical takeaway is that offset and skew are not one-time nuisances you subtract once. Offset must be re-estimated because clocks are set imperfectly; skew must be re-estimated because a fixed rate error accumulates without bound; and drift means both estimates go stale, so synchronization is a process you run continuously, not a calibration you do at the factory.

Where a timestamp really comes from

A single sensor sample can carry several different timestamps, and confusing them is the most common synchronization bug in practice. A reading acquires its capture time when the physical event is transduced, a hardware timestamp if the sensor's own clock latches the sample, an arrival timestamp when the packet reaches the host, and a host timestamp when your software finally reads it off a queue. Between capture and host there is exposure time, bus transfer, driver buffering, and operating-system scheduling, and each adds a delay that is partly fixed and partly random. The fixed part is latency; the random part is jitter, treated as a failure mode in its own right in Section 3.4.

The rule that saves projects is: timestamp as close to the physical event as the hardware allows, and record which clock did it. A camera that exposes a hardware timestamp at the middle of its integration window is worth far more than one you stamp when the USB frame lands, because the arrival time smears the true capture instant by a variable few milliseconds. On the software side, always separate the monotonic clock (a counter that only ever increases, immune to being reset) from the wall clock (calendar time, which can jump backward when NTP corrects it or a leap second lands). Use the monotonic clock to measure durations and align streams; use the wall clock only to label data with human-readable dates. Mixing them produces negative time intervals and reversed sample orders that are maddening to debug.

Mechanisms that pull sensors onto one clock

There is a ladder of synchronization techniques, and the right rung depends on how tightly your sensors must agree. From loosest to tightest:

NTP (Network Time Protocol) disciplines a host's wall clock over an ordinary network to within a few milliseconds on a LAN, tens of milliseconds over the internet. It is enough to correlate slow logs and timestamp events for humans, and nowhere near enough to fuse a 100 Hz IMU with a camera. PTP (Precision Time Protocol, IEEE 1588) uses hardware timestamping in the network interface to exchange delay-measurement messages and reach sub-microsecond agreement across devices on the same switched network; it is the backbone of synchronized industrial and automotive sensor buses. GPS with a PPS (pulse-per-second) signal gives every receiver a physical electrical edge aligned to UTC to within tens of nanoseconds, which is why GPS-disciplined clocks anchor large outdoor multi-sensor rigs and are covered further in Chapter 25. A hardware trigger is the tightest of all: one electrical line fires multiple sensors at the same instant, so they share not just a clock but a common exposure event, eliminating offset by construction.

When you cannot rewire the hardware, you fall back to post-hoc software alignment: record the best timestamps you have, then estimate the residual offset from the data itself. If two sensors observe the same physical event (a shake felt by two IMUs, an audio-visual clap), the lag that maximizes their cross-correlation is a direct estimate of \(\theta_i - \theta_j\). This is cheap, requires no special hardware, and is the workhorse for aligning consumer devices that were never designed to be synchronized. It cannot fix skew on its own, though, so for long recordings you fit the full affine map, not just a constant shift.

import numpy as np

# Paired timestamps: each row is (sensor_clock_reading, reference_clock_reading)
# gathered whenever a shared event let us observe both clocks at once.
sensor = np.array([0.001, 1.002, 2.004, 3.005, 4.007, 5.008])   # device clock (s)
ref    = np.array([0.000, 1.000, 2.000, 3.000, 4.000, 5.000])   # reference clock (s)

# Fit ref = a * sensor + b  (a undoes skew, b undoes offset)
A = np.vstack([sensor, np.ones_like(sensor)]).T
(a, b), *_ = np.linalg.lstsq(A, ref, rcond=None)

skew_ppm = (a - 1.0) * 1e6
print(f"skew = {skew_ppm:6.1f} ppm, offset = {b*1e3:6.2f} ms")

# Map any future device stamp onto the reference timeline:
def to_reference(device_stamp):
    return a * device_stamp + b

The code above is the whole idea in a dozen lines: gather pairs of readings taken at the same real instant, fit a line, and reuse it. In production you re-fit this map on a sliding window so it tracks drift rather than freezing a stale skew estimate.

import pandas as pd
# imu, cam: DataFrames sorted by a common-timeline column "t"
aligned = pd.merge_asof(cam, imu, on="t",
                        direction="nearest",
                        tolerance=0.005)   # drop pairs > 5 ms apart

The timestamp contract: what to record so alignment stays possible

Synchronization is only recoverable if you preserve the information it needs, and this is a data-engineering discipline as much as a signal one, revisited under leakage-safe pipelines in Chapter 5. Every sample should carry: the timestamp value, the clock domain it came from (monotonic versus wall, and which device), and ideally the stamping stage (capture, hardware, or arrival). Store timestamps at nanosecond resolution in integer counts, not 32-bit floats, because a float32 second-count loses millisecond precision within an hour of uptime. Never overwrite a hardware timestamp with a host one to "clean up" the record; you are destroying the very evidence a later alignment step depends on. If you log the delay-measurement exchanges from PTP or the PPS edges from GPS, you can reconstruct and audit the timebase after the fact, which is the difference between a dataset that can be re-synchronized and one that is permanently smeared.

What's Next

In Section 3.4, we confront what happens when the timeline you just built develops holes: samples that never arrive, timestamps that jitter around their true instant, and packets lost in transit. Clean synchronization assumes every sample is present and stamped; the next section drops that assumption and shows how to detect, mark, and reason about the gaps without letting them poison a downstream model.