Neuromorphic hardware (Loihi 2, Speck)

"A GPU asked me to hand it a dense tensor so it could multiply the zeros too. The neuromorphic chip just waited, silent, until a spike arrived, and then it woke only the one core that cared. I have grown fond of hardware that refuses to do work nobody asked for."

An Event-Driven AI Agent

The von Neumann wall and why event-driven silicon dodges it

What a conventional accelerator spends most of its energy on is not arithmetic but data movement. Fetching a weight from off-chip DRAM costs on the order of a hundred times more energy than the multiply-accumulate (MAC) that consumes it, so a large model is memory-bound and power-hungry regardless of how sparse its activations happen to be. This is the von Neumann bottleneck: separating the memory that holds parameters from the compute unit that uses them forces a constant, expensive commute. Why neuromorphic design attacks this is architectural. The chip is a mesh of many small neuromorphic cores, each holding a slice of neurons, their membrane state, and the synaptic weights that feed them in local SRAM, physically adjacent to the adders that update them. Computation is in-memory and event-driven: nothing happens on a core until a spike is routed to it, at which point it adds the relevant weights to its target membranes and, if a threshold is crossed, emits its own spike onto the network fabric. How this saves energy is direct: the machine's activity tracks the spike rate of the workload, so a sparse stream from an event camera (which is nearly all zeros, per Section 46.1) drives correspondingly sparse compute. Idle neurons cost leakage current and nothing more.

Loihi 2: a programmable digital neuromorphic processor

What Loihi 2 is: Intel's second-generation research chip, fabricated on the Intel 4 process and announced in 2021, packing roughly a million programmable neurons and on the order of 120 million synapses into an asynchronous mesh of 128 neuromorphic cores plus embedded microcontroller cores for management. How it advances on the original Loihi matters for model builders. First, neuron models are no longer fixed leaky-integrate-and-fire units; each core runs a tiny programmable instruction set, so you can implement custom dynamics (resonate-and-fire, adaptive thresholds, multi-compartment neurons) rather than being locked to one equation. Second, spikes carry a small integer graded payload instead of a bare binary event, which lets a spike communicate a magnitude and dramatically improves the accuracy achievable at a given spike count. Third, the chip supports on-chip synaptic plasticity, so local learning rules can adapt weights without a host in the loop, the substrate for the on-device continual learning explored in Chapter 62. Why you would reach for it: Loihi 2 is a research platform for finding algorithms that exploit temporal sparsity, and Intel has ganged 1,152 of the chips into the Hala Point system (over a billion neurons) to study neuromorphic workloads at scale. It is programmed through the open-source Lava framework, which lets you describe a network of processes and compile the same model to a CPU simulator or to physical Loihi silicon.

Speck and the always-on sensor-fused edge

What Speck represents is the opposite end of the spectrum from Hala Point: a single small chip from SynSense that integrates a dynamic-vision-sensor pixel array directly with a fully event-driven spiking convolutional processor (its DYNAP-CNN core), so events never leave the die to be serialized over a bus. Why that integration is the whole point: because the sensor and the network share the chip and both are event-driven, the system draws sub-milliwatt power when the scene is static and wakes only for the pixels that changed, giving always-on vision on a coin cell. There is no frame, no host processor, and no idle draw beyond leakage. How it is used: you train a small quantized convolutional SNN offline, then map it onto the chip's fixed pool of cores; inference then runs entirely on-sensor with millisecond, gesture-scale latency. Speck sits in a family of commercial neuromorphic edge parts worth naming: SynSense also ships Xylo for low-dimensional signals such as audio and biosignals, BrainChip sells the Akida processor, and Innatera markets an analog spiking microcontroller. The common thread is the same as Chapter 63's batteryless devices: a power budget so tight that doing nothing when nothing happens is the only viable design.

The energy model: when spikes actually beat MACs

What decides whether neuromorphic hardware wins is a single back-of-envelope comparison you can compute before touching a chip. A dense ANN layer costs \(NM\) MACs every inference regardless of activity. The spiking equivalent, run over \(T\) timesteps at average firing rate \(r\), costs roughly \(r\,T\,NM\) synaptic accumulate operations, each far cheaper per operation than a floating-point MAC. Why this is not automatically a win: if your SNN needs many timesteps \(T\) to reach acceptable accuracy, or if the firing rate \(r\) is high, the product \(rT\) can exceed one and you have more work than the dense baseline, not less. The neuromorphic advantage is real only when the workload is genuinely sparse and low-latency, exactly the regime an event camera provides. The snippet below makes the crossover explicit.

import numpy as np

def energy_ratio(N, M, r, T, e_mac_pj=3.1, e_synop_pj=0.02):
    """Energy of a spiking layer vs its dense-ANN twin, per inference.

    N, M       : input and output neuron counts (fanout NM synapses).
    r          : average firing rate (fraction of neurons active per step).
    T          : number of timesteps the SNN integrates over.
    e_mac_pj   : energy per dense float MAC (pJ), GPU-class accelerator.
    e_synop_pj : energy per neuromorphic synaptic accumulate (pJ).
    Returns (spiking_energy_pJ, dense_energy_pJ, speedup) where speedup > 1
    means the neuromorphic mapping is cheaper.
    """
    dense_pj   = N * M * e_mac_pj                 # every synapse, every time
    spiking_pj = r * T * N * M * e_synop_pj        # only active synapses fire
    return spiking_pj, dense_pj, dense_pj / spiking_pj

for r in [0.02, 0.10, 0.30]:
    s, d, gain = energy_ratio(N=1024, M=1024, r=r, T=8)
    verdict = "neuromorphic wins" if gain > 1 else "dense wins"
    print(f"r={r:>4}: spiking={s/1e3:7.1f} nJ  dense={d/1e3:6.1f} nJ  gain={gain:5.1f}x  -> {verdict}")

The energy_ratio function is the discipline this whole modality demands: it forces you to ask what firing rate and timestep count your trained SNN actually needs, because those two numbers, not the chip's peak specs, determine whether you spend milliwatts or watts. A model that scores well only by firing densely has quietly given back the reason you chose neuromorphic silicon at all.

When to reach for it, and when not to

When neuromorphic silicon is the right answer follows directly from the energy model and the chips' shapes. Reach for it when the input is genuinely sparse and event-native (an event camera, a keyword spotter, a biosignal spike train as in Chapter 31), when the power budget is measured in milliwatts and the device must be always-on, and when latency must be microseconds-to-milliseconds without a host processor in the loop. When to walk away: dense, well-lit, throughput-heavy vision belongs on a conventional accelerator; a large model that will not fit in a chip's small neuron budget belongs on a GPU; and a project on a tight schedule may not survive the immature, fragmented tooling. The clean design rule is the one the sensor itself taught in Section 46.1: neuromorphic hardware rewards you for the work you avoid, so it wins precisely where most of the data is silence.

What's Next

In Section 46.6, we address the recurring limit that surfaced throughout this section: an event stream is blind to a static scene and carries no absolute intensity, so a robust perception system rarely runs on events alone. We build the fusion of event streams, conventional frames, and inertial measurements (event-frame-IMU fusion), combining the microsecond timing of events with the absolute reference of frames and the ego-motion signal of an IMU into a single, drift-resistant estimate.