Time-series SSM variants (SiMBA, TimeMachine, Bi-Mamba)

"Mamba taught me to read a signal front to back in linear time. Then someone showed me the recording was already finished, and I had been reading it with one eye closed."

A Retrospective AI Agent

Why vanilla Mamba is not the end of the story

Recall the core Mamba recurrence, a selective linear state-space update run over the sequence:

What makes this powerful is that \(\bar A_t, \bar B_t, C_t\) depend on the input, so the model chooses what to remember. Why it needs adapting for sensing comes down to three structural mismatches. First, the recurrence is causal: \(h_t\) sees only \(x_1 \dots x_t\). That is correct for generation and for the streaming inference of Chapter 60, but wasteful when you are classifying a complete, already-captured window and the discriminative evidence sits after the moment you are encoding. Second, a single Mamba scan mixes time beautifully but says nothing about how to mix channels; treated naively, a multivariate signal is either flattened (destroying which sensor is which) or run channel-by-channel (destroying cross-sensor coupling). Third, one scan operates at one effective timescale set by \(\Delta\), yet a fault signature and a slow thermal drift want different resolutions. When these mismatches bite is precisely the sensing regime: offline windows, many correlated channels, multi-scale dynamics. Each variant below targets one mismatch head-on.

Bi-Mamba: reading the window in both directions

The cheapest fix addresses causality. A bidirectional Mamba block runs two selective scans: one over the sequence as given, one over its time-reversed copy, then recombines them so every position is informed by both its past and its future. Concretely, for input \(x \in \mathbb{R}^{L \times d}\) you compute a forward output and a backward output and fuse them:

where \(f_{\rightarrow}\) and \(f_{\leftarrow}\) are two Mamba scans with separate weights and \(\operatorname{flip}\) reverses the time axis. This is the pattern Vision Mamba (Vim) introduced for images and that Bi-Mamba-style forecasters carried into time series. Why it helps sensing: the significance of a transient often depends on what follows it. A momentary dropout in a PPG trace is benign if the pulse recovers cleanly and alarming if it precedes an arrhythmia, and only a backward pass lets the encoding of the dropout know which happened. The cost is honest and modest: two scans instead of one, so roughly double the compute and parameters of a plain Mamba layer, still linear in \(L\). The rule of thumb: use bidirectional scanning whenever the whole window is available at inference (classification, retrospective detection, forecasting from a fixed lookback) and stay strictly causal, single-direction, only when you must emit outputs as samples arrive.

import torch, torch.nn as nn
from mamba_ssm import Mamba  # pip install mamba-ssm

class BiMamba(nn.Module):
    """Bidirectional Mamba: one forward scan, one backward scan, fused."""
    def __init__(self, d_model):
        super().__init__()
        self.fwd = Mamba(d_model=d_model)   # scans x as given
        self.bwd = Mamba(d_model=d_model)   # scans time-reversed x
        self.norm = nn.LayerNorm(d_model)

    def forward(self, x):                    # x: (batch, length, d_model)
        f = self.fwd(x)
        b = self.bwd(x.flip(dims=[1])).flip(dims=[1])
        return self.norm(f + b)              # every step sees past AND future

x = torch.randn(4, 2048, 128, device="cuda")   # 4 windows, 2048 steps, 128 features
out = BiMamba(128).to("cuda")(x)
print(out.shape)   # torch.Size([4, 2048, 128]); cost is 2 scans, still O(L)

Listing 16.4 is the whole idea in a dozen lines because the heavy machinery, the hardware-aware selective scan, lives inside the library call. That is worth pausing on.

SiMBA: fixing channel mixing and deep-stack stability

Bidirectionality solves time. It does not solve channels, and it does not, on its own, make a deep Mamba stack train stably. SiMBA (Simplified Mamba-Based Architecture) tackles both. Its design principle is a clean division of labor: use Mamba for sequence mixing (across time) and a dedicated frequency-domain operator called EinFFT for channel mixing (across sensors). EinFFT transforms the channel dimension with an FFT, applies a learned complex-valued weighting via an Einstein-summation MLP, and inverts it, which mixes channels globally at \(O(d \log d)\) cost while, crucially, keeping the activations well-conditioned. That last point is the practical payoff: naive Mamba stacks were observed to grow unstable as depth increased, and SiMBA's structured spectral channel mixing was shown to tame that, letting the architecture scale to larger networks and reach state-of-the-art results on standard multivariate forecasting benchmarks. When to reach for it: many correlated channels where cross-sensor structure is diagnostic (think a full 12-lead ECG or a wide industrial tag set), and deep models where plain-Mamba training has been finicky.

TimeMachine: many Mambas for many timescales

The third gap is scale. TimeMachine attacks long-horizon multivariate forecasting by composing four Mamba modules into a nested, two-level structure that provides global and local context at multiple resolutions from a single linear-time backbone. The outer level captures long-range, coarse structure across the lookback; the inner level refines fine detail. Just as important, TimeMachine makes the channel question adaptive rather than fixed: it selects between channel-mixing and channel-independent processing based on the number of channels and the data regime, so a two-sensor stream and a hundred-sensor stream are handled differently by design instead of by a hand-tuned switch. The result keeps Mamba's linear scaling and low memory footprint while matching or beating transformer forecasters on long-context benchmarks. Why this matters for sensing specifically: real deployments span a huge channel-count range, from a single wearable strap to a plant-wide SCADA feed, and a model that auto-adapts its channel strategy is far easier to ship across that range than one you must re-architect per site.

Choosing a variant

Table 16.4 lines the three up against the mismatch each one fixes. Read it as a decision aid, not a ranking: they are complementary, and production models often combine them (a bidirectional, SiMBA-style channel-mixing block inside a multi-scale stack).

Two cross-cutting disciplines carry over from the rest of the book. Bidirectionality is off-limits for any truly streaming deployment, so decide your inference mode before you pick a variant, not after. And whichever you choose, split by recording, subject, or machine rather than by window when you evaluate, so a long bidirectional context does not quietly memorize a device it will also be tested on.

What's Next

In Section 16.5, we stop adding variants and start measuring. We put SSMs and transformers on the same bench and compare them where deployment actually decides: accuracy at long horizons, inference latency, memory footprint, and how each one fits inside the tight power and silicon budgets of an on-device sensor system.