SSMs vs transformers: accuracy, latency, memory, on-device fit

"The transformer read the entire recording again for every new sample. I kept a small vector and answered before it finished re-reading the first second."

A Frugal AI Agent

The four axes, side by side

Start with the asymptotics, because they drive everything else. For a length-\(L\) sequence, model width \(d\), and SSM state size \(N\), the dominant costs are as follows. The table entries are per-layer; multiply by depth for the full model.

Read the table as a decision tool, not a scoreboard. The \(O(L^2)\) training cost is often tolerable because you train once, offline, on windowed data. The rows that bite in production are the inference step and the memory footprint, because those are paid on the device, forever, once per incoming sample. A transformer serving a continuous stream must either cap its context (throwing away old information) or carry a key-value cache that grows linearly with how long it has been running; neither is acceptable on a coin-cell wearable. The SSM simply keeps its fixed-size state and steps forward.

Accuracy: who actually wins, and on what

On long-sequence classification and forecasting for sensor data, well-tuned SSMs and transformers land within a point or two of each other, and which one leads depends on the task structure rather than on raw capacity. Selective SSMs (Mamba, and the time-series variants of Section 16.4) tend to lead when the signal is long and its informative structure is smooth or periodic: gait cycles, respiration envelopes, slow machine degradation. Transformers tend to lead when the task hinges on retrieving a specific earlier event by its content (find the one prior beat that looked like this one), because any-to-any attention is a literal content-addressable lookup and a compressed state is not. The selection gate in Mamba narrows this gap by letting the recurrence copy or ignore inputs on demand, but it does not fully close it. The pragmatic reading is that for most streaming sensor classification you lose little or no accuracy moving to an SSM, and for the recall-heavy minority you either keep attention or use the hybrids of Section 16.6. Whichever you pick, evaluate on a leakage-safe split (Chapter 5); with long contexts it is easy to let a window straddle the train/test boundary and inflate both models equally.

Latency and memory: the axis sensing actually cares about

Latency splits into two very different questions. Throughput (samples per second during a parallel forward pass) favors whichever kernel is better tuned for your hardware; on a modern GPU with FlashAttention a transformer at short \(L\) can be extremely fast, and an SSM's advantage only appears once \(L\) is large enough for \(O(L^2)\) to dominate. Streaming step latency (time to emit one output for one new sample, with the model already caught up) is where the families diverge structurally. The SSM step is a fixed-size state update, constant in \(L\); the transformer step must attend over a cache that grows with elapsed time. For an always-on device the second number is the one that sets your battery life, and it is the one an SSM wins by construction. Memory follows the same logic: the SSM carries \(O(dN)\) numbers regardless of stream length, while the transformer's KV cache is \(O(Ld)\) and grows without bound until you evict.

The listing below measures this directly. It times a single autoregressive step for a growing context and reports how each family scales.

import time, numpy as np

d, N = 256, 16                      # model width, SSM state size

def ssm_step(state, u, Abar, Bbar, C):
    # O(d*N): fixed-size state update, independent of how long we have streamed
    state = Abar * state + Bbar * u[:, None]      # (d, N)
    return state, (state * C).sum(axis=1)         # (d,)

def attn_step(kv_cache, q):
    # O(L*d): must attend over every cached key/value seen so far
    scores = kv_cache["k"] @ q / np.sqrt(d)       # (L,)
    w = np.exp(scores - scores.max()); w /= w.sum()
    return w @ kv_cache["v"]                       # (d,)

rng = np.random.default_rng(0)
Abar, Bbar, C = rng.normal(size=(d, N)), rng.normal(size=(d, N)), rng.normal(size=(d, N))

for L in [256, 4096, 65536]:
    # SSM: state size never changes with L
    state = np.zeros((d, N)); u = rng.normal(size=(1, d))
    t0 = time.perf_counter()
    for _ in range(200): state, _ = ssm_step(state, u, Abar, Bbar, C)
    t_ssm = (time.perf_counter() - t0) / 200

    # Transformer: cache holds L past tokens
    kv = {"k": rng.normal(size=(L, d)), "v": rng.normal(size=(L, d))}
    q = rng.normal(size=d)
    t0 = time.perf_counter()
    for _ in range(200): _ = attn_step(kv, q)
    t_attn = (time.perf_counter() - t0) / 200

    print(f"L={L:6d}  SSM step={t_ssm*1e6:7.1f} us   "
          f"attn step={t_attn*1e6:8.1f} us   "
          f"cache={kv['k'].nbytes/1e6:6.1f} MB")

Run this and you see the SSM step time hold flat while the attention step time and cache size climb with \(L\). That flat line is why the same Mamba weights that trained on a workstation can step on a microcontroller: nothing about the runtime cost depends on how long the sensor has been streaming.

On-device fit: quantization, tooling, and the honest caveats

Constant memory is necessary but not sufficient for edge deployment; the model also has to quantize cleanly and run on the runtime you actually have. Here the picture is mixed and worth stating plainly. In the model's favor: the SSM's fixed state and elementwise recurrence map naturally onto streaming, fixed-point hardware, and the absence of a growing cache removes the memory-management headache that dominates transformer edge serving. Against it: the selective SSM's internal dynamics span a wide dynamic range (the state can hold information across thousands of steps), so aggressive 8-bit quantization of the recurrence needs care, and the tooling is younger. Transformer quantization and edge runtimes (the toolchain of Chapter 59 and TinyML of Chapter 61) are far more mature, with off-the-shelf converters and vendor kernels. The practical verdict: for a long-context, streaming, tightly memory-bound target, an SSM is usually the right architecture and the extra quantization effort pays for itself; for a short-context task on a platform with a polished transformer runtime, the transformer's tooling maturity can win on time-to-ship even where the SSM would be theoretically leaner.

A decision recipe

Fold the four axes into one procedure. First, fix the context length your task truly needs, not the one a benchmark used. Second, if that length is short (hundreds of samples) and the task is recall-heavy, default to a transformer and lean on its mature tooling. Third, if the length is long (thousands and up) or the target is memory-bound and always-on, default to a selective SSM and budget a little extra time for quantization. Fourth, if you need both long cheap context and sharp recall, reach for a hybrid. In every case, measure latency and memory at the real context length with the harness above, and validate accuracy on a leakage-safe split so a long window does not smuggle test information into training. The tradeoff is a measurement, not an opinion; make it one.

What's Next

In Section 16.6, we stop choosing sides. Hybrid SSM-attention architectures interleave a majority of cheap linear-time SSM layers with a few attention layers, buying the SSM's constant-memory streaming and long context together with the transformer's sharp content-based recall, and we look at how to place those attention layers so the hybrid keeps almost all of the SSM's on-device thrift.