"Two sensors that disagree about when and where are not two sensors. They are one sensor and a rumor."
A Punctual AI Agent
The big picture
Section 48.2 sorted fusion architectures into early, middle, and late. Every one of those architectures makes a silent assumption: that when you place two measurements side by side, they describe the same instant of the same physical scene. That assumption is almost never true out of the box. A camera and a lidar run off different clocks, expose at different rates, and sit a few centimeters apart in different orientations. Before any fusion rule can add value, you have to answer two questions precisely. When did each measurement happen on a common timeline (temporal alignment), and where was each sensor pointing in a common frame (calibration). Get these wrong and fusion does not merely fail to help; it actively degrades the estimate below the better single sensor. This section makes both problems concrete, shows how to measure and correct them, and tells you when a library does it for you.
This section assumes you are comfortable with sampling, timestamps, and the idea of a clock offset from Chapter 3, and with the measurement-model view of a sensor from Chapter 2. Alignment and calibration are the two coordinate transforms, one in time and one in space, that turn a bag of independent streams into something a fusion rule can trust.
Two clocks, two frames: what the problem actually is
What it is. Fusion combines measurements \(x_A\) and \(x_B\) that are supposed to describe the same world. To be comparable, each must be expressed against two shared references. The first is a common time base: a single monotonic clock on which every sample carries an honest timestamp. The second is a common spatial frame: a fixed coordinate system (often the vehicle body or a robot base link) into which every sensor's raw readings can be projected. Temporal alignment is the machinery that puts samples on the shared clock; calibration is the machinery that puts them in the shared frame.
Why it is non-negotiable. Consider a car at 20 m/s (72 km/h). A 50 ms timestamp error, easily produced by an unsynchronized USB camera, corresponds to one metre of real-world displacement. Fuse a lidar return with a camera detection that are one metre apart and you have not localized an obstacle; you have smeared it. Spatial error compounds the same way: a one-degree extrinsic rotation error between lidar and camera puts a target 35 cm off at 20 m range. Because fusion multiplies confidence, a confidently wrong association is worse than an honest single-sensor guess. This is why misalignment is the single most common reason a multi-sensor system underperforms its best individual sensor.
Key insight
Alignment error does not average out. Random measurement noise shrinks when you fuse more samples; a fixed clock offset or a miscalibrated mounting angle is a bias, and fusing more biased samples converges confidently to the wrong answer. Every fusion rule in Chapters 49 and 50 implicitly assumes alignment error is negligible next to sensor noise. Your job in this section is to make that assumption true, not to hope it is.
Where time goes wrong: offset, skew, latency, jitter
Four distinct failure modes hide inside the phrase "the clocks disagree," and they need different fixes. Model sensor \(B\)'s reported timestamp \(t_B\) against the true reference time \(t\) as
$$ t_B = t + b + s\,t + \ell + j, $$where \(b\) is a constant offset (the clocks started at different zeros), \(s\) is skew (a slow drift because no two crystal oscillators tick at exactly the same rate, typically tens of parts per million), \(\ell\) is latency (the fixed pipeline delay between the physical event and the moment a timestamp is stamped, for example exposure plus readout plus USB transport), and \(j\) is jitter (the random, sample-to-sample variation in that delay).
Why the split matters. Each term has a different remedy. Offset \(b\) and skew \(s\) are estimated once and tracked slowly, exactly what the Precision Time Protocol (PTP, IEEE 1588) and GNSS-disciplined clocks do; a pulse-per-second (PPS) signal from a GNSS receiver pins \(b\) and \(s\) to atomic time across an entire sensor rig. Latency \(\ell\) is a property of the pipeline you measure once with a known stimulus (flash an LED, look for it in the frame) and subtract. Jitter \(j\) is irreducible noise you can only bound, by timestamping as close to the sensor as possible, ideally in hardware at the moment of capture rather than at the moment your software receives the packet.
Common Misconception
"The timestamp on the message is when the measurement happened." On most buses it is when your host received the message, which includes a driver-dependent, load-dependent transport delay. Software-receive timestamps routinely carry tens of milliseconds of jitter. Trust a hardware capture timestamp; treat a receive timestamp as an upper bound corrupted by everything between the sensor and your callback.
Aligning streams in time
Once timestamps are honest, two streams at different rates still do not share sample instants. A lidar spinning at 10 Hz and an IMU at 200 Hz never line up naturally. The standard fix is resampling by interpolation: pick one stream (usually the slower or the one you predict onto) as the reference timeline, and for each reference timestamp interpolate the other stream's value there. Linear interpolation suffices for smooth quantities; for rotations you interpolate on the sphere (SLERP), a detail Chapter 24 makes precise.
But interpolation presumes you already know the offset \(b\). When you do not, and the two sensors observe a shared physical signature (both feel the same sharp motion, both see the same flash), you can estimate the delay directly by cross-correlation: slide one signal against the other and find the lag that maximizes their agreement. The code below recovers a sub-sample delay between two accelerometer streams and then resamples the second onto the first's clock.
import numpy as np
from scipy.signal import correlate, correlation_lags
# Two streams of a shared motion signature, sampled at fs Hz.
# b_true is the unknown offset (in samples) we want to recover.
fs = 200.0
t = np.arange(0, 10, 1/fs)
motion = np.sin(2*np.pi*1.5*t) + 0.4*np.sin(2*np.pi*4*t)
a = motion + 0.05*np.random.randn(t.size) # sensor A
b_true = 17 # sensor B lags A by 17 samples
b = np.roll(motion, b_true) + 0.05*np.random.randn(t.size)
# 1) Estimate the integer lag that best aligns B to A.
xcorr = correlate(b - b.mean(), a - a.mean(), mode="full")
lags = correlation_lags(b.size, a.size, mode="full")
lag_hat = lags[np.argmax(xcorr)] # -> ~17 samples
print(f"estimated offset: {lag_hat} samples = {lag_hat/fs*1e3:.1f} ms")
# 2) Undo the offset, then resample B onto A's exact timestamps.
t_b_corrected = t - lag_hat/fs # B's true event times
b_on_a_clock = np.interp(t, t_b_corrected, b) # aligned to A's grid
When to reach for cross-correlation. It works whenever the two sensors share an observable transient (a footstep felt by two IMUs, a clap heard by two microphones, a checkerboard waved in front of two cameras). When they do not, you fall back on a hardware trigger or a disciplined clock, because there is no signal to correlate.
The right tool: approximate-time synchronization in one object
In a robotics stack you rarely hand-roll the buffering, nearest-neighbour matching, and slop-tolerance logic that pairs messages from two streams. The ROS message_filters.ApproximateTimeSynchronizer collapses roughly sixty lines of ring-buffer-and-tolerance bookkeeping into three:
from message_filters import ApproximateTimeSynchronizer, Subscriber
sync = ApproximateTimeSynchronizer(
[Subscriber("/camera/image", Image), Subscriber("/lidar/points", PointCloud2)],
queue_size=10, slop=0.02) # pair messages within 20 ms
sync.registerCallback(fuse_frame) # called only with time-matched pairsmessage_filters. The synchronizer owns the per-topic buffers and the matching policy; you supply only the slop window and the fused callback. It handles arrival-order races and dropped messages that a naive "take the latest of each" loop gets wrong.Calibration: putting every sensor in one frame
Time solved, space remains. Calibration splits into two layers. Intrinsic calibration characterizes a single sensor's internal map from world to raw reading: a camera's focal length and lens distortion, an IMU's bias, scale-factor, and axis non-orthogonality. Extrinsic calibration is the rigid-body transform \(T_{A\leftarrow B} \in SE(3)\), a rotation \(R\) and translation \(t\), that expresses sensor \(B\)'s frame in sensor \(A\)'s frame, so a point \(p_B\) becomes \(p_A = R\,p_B + t\).
How you get the extrinsics. Target-based calibration shows both sensors a known object (a checkerboard for cameras, a board with a reflective pattern for camera-lidar) and solves for the transform that makes the observations agree. Targetless methods, increasingly the norm, exploit motion: if you know each sensor's own trajectory over time, the classic hand-eye equation \(AX = XB\) recovers the fixed transform \(X\) between them from the motions \(A\) and \(B\) they each report. This is the calibration backbone that SLAM systems and lidar pipelines rely on.
In practice: an autonomous shuttle that drifted one winter
A low-speed autonomous shuttle fused a roof lidar with a forward camera to place pedestrians. It passed every summer test. In January, pedestrian boxes began landing half a metre to the left of the true position, enough to misjudge a curbside crossing. Nothing in the perception model had changed. The cause was thermal: the aluminium roof rack contracted in the cold, rotating the lidar about 0.4 degrees relative to the camera. That tiny extrinsic error, invisible on any single sensor, became a 30 cm fusion bias at typical detection range. The fix was twofold: add an online extrinsic-consistency check that compares lidar-projected edges against camera edges every few seconds and flags drift, and re-estimate the transform continuously instead of trusting a factory calibration frozen in a warm garage. The episode is the whole section in miniature: alignment is not a one-time setup step, it is a quantity that drifts and must be monitored like any other.
Keeping alignment honest over a system's life
Why one-time calibration is not enough. Offsets drift as crystals age, extrinsics shift with temperature and vibration, and latencies change when a software update reorders a pipeline. A calibration is a measurement, and like every measurement it comes with uncertainty and a shelf life. Production systems therefore treat alignment as a monitored, versioned artifact: calibration parameters live in configuration with timestamps and covariances, an online consistency check watches for drift (do the two sensors still agree on a shared feature?), and a recalibration routine either runs continuously or triggers when the check trips. Chapter 49 shows how a well-run filter exposes drift automatically, because a growing innovation between two sensors is exactly the fault signal a consistency test is built to catch.
When to invest how much. If your sensors run off one hardware clock and are rigidly co-mounted (a phone's camera and IMU), factory intrinsics plus a hardware timestamp may carry you. If they are physically separate, thermally exposed, or field-serviceable (a vehicle, a robot arm, a distributed sensor network), budget for online estimation from the start. The cost of getting this wrong is not a crash you can debug; it is a quiet, confident bias that erodes every downstream number.
Exercise
Take two phones (or two IMU logs you can record) and clap or tap them together sharply three times.
- Extract the acceleration magnitude from each and use the cross-correlation code above to estimate the constant offset \(b\) between their clocks. How many milliseconds is it, and how stable is the estimate across the three taps?
- Now compute the offset using only the first tap versus using the full recording. If they differ, what does that tell you about skew \(s\) versus offset \(b\)?
- At a walking speed of 1.4 m/s, how far would your measured offset displace a fused position estimate? At 20 m/s?
Hint
If the offset estimated from the last tap differs from the offset estimated from the first tap, the gap divided by the elapsed time is a direct estimate of skew \(s\) in parts per million. A single offset correction cannot fix a stream whose skew is significant; you need to re-estimate or discipline the clock.
Self-check
- Explain why a constant 30 ms clock offset is more dangerous to fusion than 30 ms of random timestamp jitter of the same magnitude.
- Distinguish intrinsic from extrinsic calibration, and give one physical parameter that belongs to each.
- Your camera-lidar system passes calibration in the lab but produces biased fused detections in the field only on hot days. Which of offset, skew, latency, or extrinsics is the likeliest culprit, and how would you confirm it online?
What's Next
In Section 48.4, we drop the comfortable assumption that every stream is present at every instant. Sensors drop out: a camera blinds in glare, a GPS fix vanishes in a tunnel, a wearable electrode peels off. We look at how a fusion system degrades gracefully when a modality goes missing, so that losing one input costs you accuracy rather than the whole estimate.