AdvancedTemporal and Sequential Networks

State space models (SSM / S4)

Structured State Space Models

Sequence layers built on the continuous linear state space model, discretized in time and computable as a fast recurrence at inference or as a long parallel convolution in training, with a special initialization (HiPPO) to preserve long history.

Overview

NameState space models – Structured State Space Models
Year introduced2022
ResearchersAlbert Gu, Karan Goel, Christopher Ré
InstitutionStanford University
Original paperEfficiently modeling long sequences with structured state spaces arXiv
Data typeSequences and time series
DomainsTime series forecasting, Audio and speech, Natural language processing

Problem it solves

The Transformer's cost is quadratic in sequence length, and RNNs are slow to train and forget distant history. Researchers needed a model that handles thousands of steps efficiently.

Background and evolution

What came before: Recurrent neural network, Temporal convolutional networks, HiPPO

  1. 2020

    HiPPO introduced a framework for initializing the state matrix to summarize history accurately. [ref]

  2. 2022

    S4 introduced a structured parameterization of matrix A that made computation efficient and achieved strong results on Long Range Arena. [ref]

  3. 2023

    S5 simplified the layer using a parallel scan for a multi-input system. [ref]

  4. 2024

    A survey of state space models as an alternative to the Transformer. [ref]

Core idea

The system $x'=Ax+Bu$, $y=Cx+Du$ is discretized with step Δ and becomes a linear recurrence. Because it is linear and time-invariant, the recurrence can be unrolled into a convolution kernel $\bar K=(C\bar B, C\bar A\bar B, \dots)$ applied to the whole sequence at once (via FFT). Initializing A with HiPPO makes the state a compressed representation of history.

Example: To classify an audio file of 16 thousand samples, an S4 layer processes the whole sequence with a long convolution, while a Transformer needs a $16000^2$ attention matrix.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
Continuous SSM (A, B, C, D)Continuous linear model
DiscretizationZOH or bilinear with step Δ
HiPPO initializationInitializing A to preserve history
Recurrent modeFor step-by-step inference
Convolutional modeFor parallel training
Nonlinearity + mixingBetween layers (GELU, linear)

Equations

1. Continuous model

$$x^{\prime}(t)=Ax(t)+Bu(t),\qquad y(t)=Cx(t)+Du(t)$$

Symbols
$x(t)$State of dimension N
$u(t)$Input
$A,B,C,D$Parameters
Dimensions
$A\in\mathbb{R}^{N\times N}$
Numerical example
For a scalar state with $A=-1$, $B=1$ and constant input 1: $x$ approaches 1.
Relation to the architecture
The theoretical basis of the layer.

2. ZOH discretization

$$\bar A=\exp(\Delta A),\quad \bar B=(\Delta A)^{-1}\left(\exp(\Delta A)-I\right)\Delta B,\quad x_k=\bar Ax_{k-1}+\bar Bu_k,\; y_k=Cx_k$$

Symbols
$\Delta$Learned time step
Dimensions
Same dimensions as A and B
Numerical example
$A=-1,\;\Delta=0.1$ ⇒ $\bar A=e^{-0.1}\approx0.905$, $\bar B\approx0.095$.
Relation to the architecture
Turns the continuous model into a recurrence.

3. Convolution form

$$\bar K=\left(C\bar B,\;C\bar A\bar B,\;\dots,\;C\bar A^{L-1}\bar B\right),\qquad y=\bar K*u$$

Symbols
$L$Sequence length
Dimensions
$\bar K\in\mathbb{R}^{L}$
Numerical example
In the previous example with $C=1$: $\bar K\approx(0.095,0.086,0.078,\dots)$.
Relation to the architecture
Allows parallel training via FFT at a cost of $O(L\log L)$.

How it works, step by step

  1. The system parameters are discretized with step Δ.
  2. The long convolution kernel is computed.
  3. The convolution is applied to the sequence via FFT.
  4. Du is added, then activation and channel mixing.
  5. At inference, the recurrence can be used at constant cost per step.

Strengths and limitations

Strengths

  • Near-linear cost in length.
  • Strong long-term memory on long-context benchmarks.
  • Two modes: recurrent for inference and convolutional for training.

Limitations

  • Linear time-invariant versions cannot choose what to remember based on content (addressed by Mamba).
  • The full S4 implementation is mathematically complex.

When to use it, and when not?

Good for

  • Very long sequences (raw audio, long series).

Not suited for

  • Tasks that rely on exact copying and retrieval from context, where attention or selective SSMs win.

Applications

DomainUse
Time series forecastingForecasting long series.
Audio and speechRaw audio modelling.
Natural language processingLong-context language modelling.
Medical image analysisLong ECG and EEG signals.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

The original S4 uses a complex A matrix with HiPPO initialization and a low-rank representation; here is a simplified real diagonal version in the S4D style to illustrate discretization and convolution.

Python (PyTorch)
# State Space Model (simplified S4D): diagonal A matrix and convolution form
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
class S4DLayer(nn.Module):
    """Real diagonal SSM per channel: ZOH discretization, then a long convolution kernel applied via FFT."""
    def __init__(self, H, N=16):
        super().__init__()
        self.log_dt = nn.Parameter(torch.rand(H) * (torch.log(torch.tensor(0.1)) - torch.log(torch.tensor(0.001))) + torch.log(torch.tensor(0.001)))
        self.log_A = nn.Parameter(torch.log(0.5 * torch.ones(H, N) + torch.arange(N)))   # A = -exp(log_A) < 0 (stable)
        self.B = nn.Parameter(torch.ones(H, N))
        self.C = nn.Parameter(torch.randn(H, N) * 0.5)
        self.D = nn.Parameter(torch.ones(H))
    def kernel(self, L):
        dt, A = self.log_dt.exp()[:, None], -self.log_A.exp()               # H×1, H×N
        Abar = torch.exp(dt * A)
        Bbar = (Abar - 1) / A * self.B                                       # ZOH for a diagonal matrix
        powers = Abar[:, :, None] ** torch.arange(L)                         # H×N×L
        return torch.einsum('hn,hnl->hl', self.C * Bbar, powers)             # K̄ = C Ā^k B̄
    def forward(self, u):                                                    # u: B×H×L
        L = u.shape[-1]
        K = self.kernel(L)
        y = torch.fft.irfft(torch.fft.rfft(u, 2 * L) * torch.fft.rfft(K, 2 * L), 2 * L)[..., :L]
        return y + self.D[:, None] * u

class SSMNet(nn.Module):
    def __init__(self, H=16, n_classes=2):
        super().__init__()
        self.inp = nn.Linear(1, H)
        self.layers = nn.ModuleList([S4DLayer(H), S4DLayer(H)])
        self.mix = nn.ModuleList([nn.Linear(H, H), nn.Linear(H, H)])
        self.out = nn.Linear(H, n_classes)
    def forward(self, x):                                                    # B×T×1
        h = self.inp(x)
        for ssm, mix in zip(self.layers, self.mix):
            h = h + mix(F.gelu(ssm(h.transpose(1, 2)).transpose(1, 2)))
        return self.out(h.mean(1))                                     # mean pooling over time (as in S4 classification)

# 3) Prepare the data
# Memory task: the signal (±1) appears only at the first step, then noise; class = initial signal
def make(n, T=200):
    y = torch.randint(0, 2, (n,))
    X = torch.randn(n, T, 1) * 0.2
    X[:, 0, 0] = y.float() * 2 - 1
    return X, y
X_train, y_train = make(800)
X_test, y_test = make(200)

model = SSMNet()

# 4) Loss function and optimizer
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=5e-3)

# 5) Training loop with the forward pass
EPOCHS, BATCH = 15, 32
for epoch in range(EPOCHS):
    model.train()
    perm = torch.randperm(len(X_train))
    total = 0.0
    for i in range(0, len(X_train), BATCH):
        idx = perm[i:i + BATCH]
        xb, yb = X_train[idx], y_train[idx]
        out = model(xb)                       # Forward Pass
        loss = loss_fn(out, yb)
        opt.zero_grad()
        loss.backward()                    # Backpropagation
        opt.step()
        total += loss.item() * len(idx)
    if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
        print(f"epoch {epoch:3d}  loss={total / len(X_train):.4f}")

# 6) Evaluation
model.eval()
with torch.no_grad():
    xb = X_test
    out = model(xb)
    acc = (out.argmax(dim=-1) == y_test).float().mean().item()
    print(f"test accuracy = {acc:.3f}")

# 7) Inference on a new sample
with torch.no_grad():
    xb = X_test[:1]
    out = model(xb)
    print("predicted class:", out.argmax(dim=-1).item())

This code was run on 2026-11-20 in 5.8 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

test accuracy = 1.000

Trainable parameters in the example: model = 2,210.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Advanced2023

Mamba: selective state space models

Mamba (Selective State Space Models)

An SSM in which B, C and the step Δ are functions of the input (selective), so the layer decides what to remember and what to ignore based on content, with a hardware-aw…

Temporal and Sequential Networks

Basic2018

Temporal convolutional networks

Temporal Convolutional Network (TCN)

A causal 1D convolutional network (it cannot see the future) with dilated convolutions and residual connections, which processes sequences in parallel and covers a long…

Temporal and Sequential Networks

Basic1997

Long short-term memory

Long Short-Term Memory (LSTM)

A recurrent network with a memory cell and gates (forget, input, output) that control what is kept and what is erased, so it learns long-term dependencies far better tha…

Temporal and Sequential Networks

Basic2017

Transformer

Transformer

An architecture based entirely on multi-head self-attention instead of recurrence or convolution, so it processes all sequence positions in parallel and links any two po…

Transformer Architectures

Research ideas

  • Evaluating S4/S5 on long overnight EEG series against TCN and Transformer with an equal memory budget.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    Why can a linear time-invariant SSM layer be trained in parallel?

  2. Mathematical

    With $A=-2$ and $\Delta=0.5$, what is $\bar A=e^{\Delta A}$?

  3. Concept

    What is the role of HiPPO initialization in S4?

References

  1. Gu, A., Goel, K., & Ré, C. (2022). Efficiently modeling long sequences with structured state spaces. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2111.00396

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  2. Gu, A., Dao, T., Ermon, S., Rudra, A., & Ré, C. (2020). HiPPO: Recurrent memory with optimal polynomial projections. Advances in Neural Information Processing Systems (NeurIPS) 33. https://arxiv.org/abs/2008.07669

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  3. Smith, J. T. H., Warrington, A., & Linderman, S. W. (2023). Simplified state space layers for sequence modeling. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2208.04933

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  4. Wang, X., Wang, S., Ding, Y., Li, Y., Wu, W., Rong, Y., et al. (2026). State space model for new generation network alternative to transformers: A survey. Visual Intelligence, 4(1), Article 29 (2026; arXiv preprint 2404.09516, 2024). https://doi.org/10.1007/s44267-026-00133-1

    DOIarXivAutomatically verified via Crossref – 2026-11-20