Event representations (voxel grids, time surfaces)

"You handed me a million timestamps and asked for a picture. Fine, but do not blame me when I flatten your favorite dimension to make it fit."

A Reluctantly Rasterizing AI Agent

This section assumes you can read a raw event stream and understand why polarity and microsecond timestamps exist, which is the sensor story from Section 46.1. It also leans on the sampling-and-time intuition from Chapter 3: an event stream is an irregularly sampled signal, and every representation here is a resampling choice with an aliasing budget. The goal is to feed the neural representations of Chapter 13 without silently discarding information.

The stream, and why it resists tensors

Formally, a window of events is a set \(\mathcal{E} = \{(x_k, y_k, t_k, p_k)\}_{k=1}^{N}\), where \(p_k \in \{-1, +1\}\) is the sign of the log-intensity change and \(N\) varies from a few hundred (a still scene) to several million per second (a textured scene under fast motion). Three properties make this awkward for standard networks. It is sparse: most pixels are silent in any short window. It is asynchronous: there is no shared clock tick, so no natural "frame". And it is temporally precise: the information often lives in the exact ordering and spacing of events, not in their spatial pattern alone. A representation is a function \(R: \mathcal{E} \to \mathbb{R}^{C \times H \times W}\) that fixes the shape while deciding, explicitly, how much of that temporal structure to keep. Think of it as the event-vision analogue of choosing a spectrogram window in Chapter 7: the transform is lossy, and the loss is the design.

Event frames: the honest baseline

The simplest representation drops time entirely inside the window. An event frame (or two-channel histogram) accumulates counts per pixel:

You get a familiar 2D image (often two channels, one per polarity, or a single signed difference) that any pretrained CNN can consume. It is fast, trivially parallel, and works surprisingly well for low-speed classification. Its weakness is exactly its simplicity: two events one microsecond apart and two events ten milliseconds apart land in the same bin, so all motion-timing information is destroyed. Under fast motion the frame also smears, because every event in the window is painted at its own stale location. Event frames are the right choice when the downstream task is appearance-dominated and the window is short; they are the wrong choice when timing carries the signal, which is most of the reason event cameras exist.

Time surfaces: recency as a feature

A time surface (also Surface of Active Events, SAE) keeps the single most recent event per pixel and encodes how recently it fired through an exponential decay. At query time \(t\),

where \(t_{\text{last}}(x,y)\) is the timestamp of the last event at that pixel and \(\tau\) is a decay constant. Freshly active pixels read near 1, stale ones fade toward 0, so the surface encodes a smooth "motion history" whose gradient points along the direction of edge travel. The constant \(\tau\) is the one knob that matters: too small and the surface is noisy and empty between events; too large and fast motion saturates it into a blur. Time surfaces are the backbone of the HOTS and HATS descriptors and of many low-power classifiers, because a single surface update is one memory write and one exponential, cheap enough for the edge budgets of Chapter 59. Their weakness is that keeping only the latest event per pixel discards event density, so a bright flickering edge and a single passing edge can look identical.

Voxel grids: giving time back a few dimensions

A voxel grid refuses the binary choice by discretizing the temporal window into \(B\) bins and distributing each event's polarity across the two nearest bins with a linear (bilinear-in-time) kernel. Normalize each timestamp to \(t^{*}_k = (B-1)\,(t_k - t_0)/(t_1 - t_0)\), then accumulate

for bin index \(b \in \{0, \dots, B-1\}\). The result is a \(B \times H \times W\) tensor: a short stack of polarity-weighted slices that a 2D CNN reads as extra input channels or a 3D CNN reads as a tiny volume. The linear splatting is what makes it differentiable and anti-aliased, so gradients flow back to timestamps and you avoid the hard binning artifacts of a naive histogram-over-time. Voxel grids with \(B\) between 5 and 15 are the standard input to optical-flow networks (EV-FlowNet), to reconstruction models, and to the recurrent transformer backbones you will meet in Section 46.3. They keep enough timing to recover motion direction and enough density to distinguish strong from weak edges, at a memory cost of \(B\) times an event frame. Notice the deep similarity to the spatiotemporal point-cloud voxelization of Chapter 42: an event stream is a point cloud in \((x, y, t)\), and a voxel grid is exactly the occupancy-grid trick applied with time as the third axis.

import numpy as np

def to_voxel_grid(xs, ys, ts, ps, H, W, B=5):
    """Bilinear-in-time voxel grid: (B, H, W) from an event window."""
    vox = np.zeros((B, H, W), dtype=np.float32)
    t0, t1 = ts[0], ts[-1]
    t_star = (B - 1) * (ts - t0) / max(t1 - t0, 1e-9)   # normalize to [0, B-1]
    lo = np.floor(t_star).astype(int)
    w_hi = t_star - lo                                   # linear split weight
    pol = ps.astype(np.float32)                          # +1 / -1
    for b_off, w in ((0, 1.0 - w_hi), (1, w_hi)):        # two nearest bins
        b = lo + b_off
        m = (b >= 0) & (b < B)                           # keep valid bins
        np.add.at(vox, (b[m], ys[m], xs[m]), pol[m] * w[m])
    return vox

# 3 events on one pixel, spread across the window -> spread across bins
xs = np.array([10, 10, 10]); ys = np.array([20, 20, 20])
ts = np.array([0.0, 5.0, 10.0]); ps = np.array([1, 1, -1])
print(to_voxel_grid(xs, ys, ts, ps, H=64, W=64, B=5)[:, 20, 10])
# -> [ 1.  0.  1.  0. -1.]  (early +, mid +, late - land in distinct bins)

The snippet above is the whole idea in a dozen lines, and it is worth writing once by hand to feel where the timing goes. In production you rarely maintain it yourself.

import tonic.transforms as T
voxel = T.ToVoxelGrid(sensor_size=(W, H, 2), n_time_bins=5)(events)
surface = T.ToTimesurface(sensor_size=(W, H, 2), tau=20000)(events)

Choosing, and letting the network choose

The decision rule follows the loss axes. If labels depend on appearance and the scene is slow, an event frame is cheapest and fine. If you need motion timing on a microwatt budget, a time surface is the frugal choice. If you are feeding a modern flow, depth, or detection network and can afford a few channels, a voxel grid is the default and the one most published results assume. A subtler option is to learn the representation: the Event Spike Tensor (EST) of Gehrig et al. replaces the fixed decay and binning kernels with a small trainable network, letting the aggregation adapt to the task, and it set the reference for treating the representation as part of the model rather than a fixed preprocessing step.

What's Next

In Section 46.3, we push past fixed representations entirely. We reconstruct intensity video directly from events with E2VID, then meet recurrent vision transformers (RVT) that consume voxelized events and carry temporal state inside the network, so the representation and the model stop being separate design decisions.