Event-to-video and recurrent vision transformers (E2VID, RVT)

"Give me a firehose of brightness changes and I can hand you back a movie, or I can skip the movie and just tell you where the car is. The second answer arrives twelve milliseconds sooner, and nobody was going to watch the movie anyway."

A Pragmatic AI Agent

This section builds directly on the event representations of Section 46.2: the voxel grids and time surfaces that turn a stream of events into a dense tensor a network can read. It assumes the recurrent and gated-memory models of Chapter 14 and the attention mechanisms of Chapter 15, because the two systems here are, at heart, a recurrent U-Net and a recurrent transformer. Our task is to explain why recurrence is not optional, what each architecture keeps and discards, and how to choose between reconstruct-then-perceive and perceive-directly.

E2VID: turning events back into video

The event stream carries, in principle, everything needed to reconstruct intensity. Each event fires when the log-brightness of a pixel crosses a threshold, so events are essentially samples of the temporal gradient \(\partial (\log I) / \partial t\). Integrate them and you recover brightness up to an unknown constant. Naive per-pixel integration drifts and smears badly because the threshold is noisy and the initial value is unknown, so E2VID (Rebecq, Ranftl, Koltun and Scaramuzza, 2019) learns the integration instead. Its network is a recurrent U-Net: at each step it consumes a voxel-grid tensor built from a short window of events, passes it through an encoder-decoder with skip connections, and, critically, threads a ConvLSTM recurrent state through the bottleneck. That state is what lets the reconstruction accumulate: a pixel that has not fired recently keeps the brightness the network inferred for it several windows ago, so static regions stay filled in even though events only report change.

The training data problem is acute, because you cannot collect ground-truth video and perfectly synchronized events at high speed for arbitrary scenes. E2VID sidesteps it by training almost entirely in simulation: the ESIM event-camera simulator renders synthetic scenes, emits physically plausible events, and provides the true intensity frames as targets. This is the synthetic-to-real strategy of Chapter 55 applied to a sensor where real labels are essentially unobtainable, and it works well enough that E2VID reconstructions transfer to real event cameras with no fine-tuning. The payoff is dramatic on exactly the scenes that break normal cameras: because events have microsecond timing and roughly 120 dB of dynamic range, E2VID reconstructs sharp video through motion that would blur a frame camera and across a tunnel exit that would saturate one, echoing the high-dynamic-range advantages discussed for thermal imaging in Chapter 45.

The code below makes the role of recurrent state concrete with a deliberately tiny, pure-numpy stand-in for the E2VID idea. It is not the trained network; it is a leaky recurrent accumulator that shows why carrying state across windows is what fills in the scene between sparse event slices.

import numpy as np

def recurrent_reconstruct(event_windows, leak=0.02, gain=0.15):
    """event_windows: list of (H, W) signed event-count maps (ON minus OFF).
    Returns a list of reconstructed log-intensity frames.
    A ConvLSTM in real E2VID replaces this leaky integrator with a learned one."""
    H, W = event_windows[0].shape
    state = np.zeros((H, W), dtype=np.float32)   # the recurrent memory
    frames = []
    for ev in event_windows:
        state = (1.0 - leak) * state + gain * ev # carry past brightness, add new change
        frames.append(state.copy())
    return frames

# synthetic: a bright square drifts right; only its edges emit events
H = W = 32
windows = []
for t in range(8):
    ev = np.zeros((H, W), np.float32)
    x = 6 + t                                    # leading edge turns pixels ON
    ev[10:22, x] = +1.0
    ev[10:22, x - 6] = -1.0                      # trailing edge turns them OFF
    windows.append(ev)

frames = recurrent_reconstruct(windows)
print("filled pixels at t=0:", int((frames[0] > 0.05).sum()),
      "  at t=7:", int((frames[7] > 0.05).sum()))

As the printed counts show, the reconstruction is nearly empty at the first window (only the initial edge has fired) and fills the square's interior as state accumulates, exactly the behavior that makes recurrence indispensable. FireNet, a lighter follow-up, keeps this recurrent skeleton but shrinks it enough to run on constrained hardware, trading a little reconstruction quality for the edge budgets of Chapter 59.

RVT: skip the picture, keep the memory

Reconstruction is a detour if the goal is a bounding box rather than a viewable frame. RVT, the Recurrent Vision Transformer of Gehrig and Scaramuzza (2023), takes the direct route: it reads event representations and outputs object detections without ever forming an image a person would recognize. Its backbone is a stack of stages, and each stage does two things in sequence. First it applies local, window-based self-attention (in the spirit of the efficient transformers you met in Chapter 15), which mixes information within small spatial neighborhoods cheaply. Then it passes the stage's features through a convolutional LSTM that carries a per-stage recurrent state forward in time. That second half is the whole point: a single event slice is far too sparse to detect anything reliably, so the LSTM remembers where objects were and what the scene looked like across many slices, letting a nearly empty current input still yield a confident, stable detection.

The result is a system that plays directly to the event camera's strengths instead of spending them. On the Gen1 and 1-megapixel automotive detection benchmarks, RVT reaches accuracy competitive with far larger prior models while running at roughly twelve milliseconds per forward step with a small parameter count, because it never densifies the stream into full frames and never pays a reconstruction network's cost. Where E2VID's philosophy is "make events look like the images our models already understand," RVT's is "teach the model to understand events, and give it memory so sparsity stops hurting." The recurrent state is the shared ingredient; what differs is whether the network's output is a picture or a decision.

Choosing between reconstruct and perceive-directly

The decision reduces to what the output is for. Reconstruct with E2VID when a human will watch the result (surveillance review, cinematography, debugging), when you must reuse a mature pretrained frame model that would be costly to retrain on events, or when several downstream tasks all want an image and a shared reconstruction amortizes the cost. Perceive directly with an RVT-style model when the output is a decision (detection, tracking, classification) and the whole reason you chose an event camera was low latency and low power, both of which reconstruction spends. A useful rule of thumb: every millisecond and every milliwatt that reconstruction consumes is deducted from the very budget the event camera was meant to protect, so reconstruct only when the bridge to frame-based tooling is worth more than the sensor's native advantages. The next sections push further in the direct-processing direction, first into spiking networks that stay asynchronous end to end, then onto neuromorphic silicon that runs them.

What's Next

In Section 46.4, we drop the assumption that perception must happen in synchronous network steps at all. Spiking neural networks process events as they arrive, keeping computation asynchronous and event-driven from sensor to decision, which sets up the neuromorphic hardware that runs them natively and closes the latency-and-power loop this section kept invoking.