Event-frame-IMU fusion

"The event stream told me exactly when the world moved, the frame told me what grey it was, and the IMU told me how far I had turned while both of them were still arguing about it. My job was to make all three agree on a single clock they never shared."

A Triangulating AI Agent

The complementary triangle

What each modality contributes is best read as a table of who covers whom. The event stream gives microsecond-latency motion cues with no blur and 120 dB of dynamic range, but carries no absolute intensity and dies when nothing moves. The intensity frame gives dense texture and absolute grey values that anchor recognition and loop closure, but blurs under fast motion and saturates in glare. The IMU gives metric scale, a gravity direction, and a high-rate motion prior (200 Hz to 1 kHz) that never blinks, but its double integration turns small bias errors into runaway position drift within seconds. Why fuse rather than pick one: the union covers the whole operating envelope. When the frame blurs, the events stay sharp; when the events fall silent because the platform paused, the frame and the still-integrating IMU hold the pose; when both cameras are starved of texture in a blank corridor, inertial dead reckoning bridges the gap until features return.

How the hardware makes this practical is the DAVIS sensor (Dynamic and Active-pixel VIsion Sensor): iniVation's DAVIS346 places an event circuit and a conventional active-pixel-sensor frame readout behind the same pixel array, and mounts an IMU on the same board with a shared hardware clock. Because events and frames come from identical optics and one timestamp source, you skip the extrinsic-calibration and clock-drift nightmare that plagues a bolted-together multi-camera rig (Chapter 48). That single-aperture design is why event-frame-IMU fusion is a real product category and not just a paper idea.

Time is the hard part: asynchronous measurements

What makes this fusion genuinely harder than stacking cameras is that the three streams do not share a rate. Events arrive asynchronously at microsecond resolution, the IMU ticks at a fixed few hundred hertz, and frames land at 20 to 60 Hz. The Kalman filter of Chapter 9 assumes a state defined at discrete synchronized instants; here no two measurements share an instant. Why this is not a formality: if you naively bin events into pseudo-frames to force synchrony, you throw away the temporal precision that was the entire reason to use an event camera, and you reintroduce the motion blur you paid to avoid.

How the field solves it comes in two flavors. The first is a continuous-time state: represent the camera trajectory as a smooth function of time, typically a cumulative cubic B-spline over \(SE(3)\), so the pose is queryable at the exact timestamp of any measurement, event or frame or inertial. A single event at time \(t_k\) then contributes a residual evaluated at \(T(t_k)\), no binning required. The second is IMU preintegration (Forster and colleagues, 2017): rather than re-integrate the inertial stream every time the optimizer nudges a pose, you pre-summarize all IMU samples between two keyframes into a single relative-motion factor with an analytic covariance, so the high-rate inertial data becomes one cheap constraint in a factor graph (Chapter 11). Both hinge on getting the timestamps right in the first place, which is why hardware-synchronized capture and the clock-alignment methods of Chapter 3 are load-bearing, not optional.

Fusion architectures, from warping to factor graphs

The classic and still-instructive move is IMU-aided motion compensation. Because a blurry event image is just events smeared along the motion they were generated by, and the IMU tells you the camera's angular velocity almost instantly, you can warp each event backward along the rotation the gyroscope measured to the moment it fired, collapsing a motion-smeared cloud into a sharp edge image. This is a cheap, feed-forward way to let the IMU sharpen the events before any feature extraction, and it is the seed of the contrast-maximization family of methods. The snippet below performs exactly this rotational warp.

import numpy as np

def imu_warp_events(xy, t, omega, K, t_ref):
    """Warp events to a reference time using the IMU's angular velocity.

    xy    : (N, 2) event pixel coordinates [x, y].
    t     : (N,) event timestamps (seconds).
    omega : (3,) constant body angular velocity from the gyro (rad/s).
    K     : (3, 3) camera intrinsics.
    t_ref : reference timestamp to warp every event onto.
    Returns (N, 2) motion-compensated pixel coordinates.
    """
    Kinv = np.linalg.inv(K)
    ones = np.ones((xy.shape[0], 1))
    rays = (Kinv @ np.hstack([xy, ones]).T).T          # back-project to bearing rays
    warped = np.empty_like(xy)
    for i, (r, dt) in enumerate(zip(rays, t_ref - t)):
        th = omega * dt                                # small-angle rotation vector
        wx, wy, wz = th
        R = np.array([[1, -wz, wy], [wz, 1, -wx], [-wy, wx, 1]])  # 1st-order SO(3)
        p = K @ (R @ r)
        warped[i] = p[:2] / p[2]
    return warped

K = np.array([[200., 0., 160.], [0., 200., 120.], [0., 0., 1.]])
xy = np.array([[170., 120.], [175., 121.], [180., 122.]])   # a smeared edge
t  = np.array([0.000, 0.002, 0.004])
omega = np.array([0.0, 0.8, 0.0])                            # 0.8 rad/s pan
print(imu_warp_events(xy, t, omega, K, t_ref=0.0).round(2))

Motion compensation is feed-forward and does not estimate a pose. The tightly coupled systems that do treat every modality as a residual in one optimizer. The landmark is Ultimate SLAM (Vidal, Rebecq, Horstschaefer, and Scaramuzza, IEEE RA-L 2018), which tracks features on both motion-compensated event images and standard frames, then fuses those visual tracks with preintegrated IMU factors in a keyframe optimizer, reporting large accuracy gains over frame-plus-IMU alone in high-speed and HDR scenes. Feature tracking across the two visual modalities is itself a research thread: EKLT (Gehrig and colleagues, 2020) tracks a frame-defined patch asynchronously through the event stream, giving the optimizer sub-frame-rate updates. The common shape is the factor graph: nodes are poses and velocities and IMU biases; edges are visual reprojection factors from events and frames plus preintegrated inertial factors; a smoother solves for the whole trajectory. This is the same machinery as visual-inertial SLAM in Chapter 52, with events added as one more factor type.

Learned fusion and where it is heading

The classical pipeline hand-designs every stage: warp, track, optimize. The current research push replaces stages, or the whole pipeline, with learned components. What is new is end-to-end event-inertial odometry networks that ingest an event representation (voxel grid or time surface, Section 46.2) alongside raw IMU samples and regress relative pose directly, with recurrent or transformer temporal cores from Chapter 14. Why this is attractive on the edge is that a single learned model can absorb the timestamp bookkeeping and the modality weighting that the classical stack splits across hand-tuned modules, and it degrades more gracefully when one modality drops out, which connects to the missing-modality robustness ideas of Chapter 50. When the classical estimator still wins is whenever you need certified, interpretable uncertainty and loop closure, which the factor graph gives you for free and the black box does not.

What's Next

In Section 46.7, we step back from any single fusion recipe to weigh the modality as a whole: the genuine low-latency, low-power, high-dynamic-range advantages that make event sensing compelling at the always-on edge, set squarely against its limits in cost, tooling maturity, noise, and the sheer scarcity of trained engineers. It is the chapter's closing argument about when to reach for an event camera and when a boring frame sensor is still the right tool.