State space models (SSM / S4)
Structured State Space Models
Sequence layers built on the continuous linear state space model, discretized in time and computable as a fast recurrence at inference or as a long parallel convolution in training, with a special initialization (HiPPO) to preserve long history.
Overview
| Name | State space models – Structured State Space Models |
|---|---|
| Year introduced | 2022 |
| Researchers | Albert Gu, Karan Goel, Christopher Ré |
| Institution | Stanford University |
| Original paper | Efficiently modeling long sequences with structured state spaces arXiv |
| Data type | Sequences and time series |
| Domains | Time series forecasting, Audio and speech, Natural language processing |
Problem it solves
The Transformer's cost is quadratic in sequence length, and RNNs are slow to train and forget distant history. Researchers needed a model that handles thousands of steps efficiently.
Background and evolution
What came before: Recurrent neural network, Temporal convolutional networks, HiPPO
- 2020
HiPPO introduced a framework for initializing the state matrix to summarize history accurately. [ref]
- 2022
S4 introduced a structured parameterization of matrix A that made computation efficient and achieved strong results on Long Range Arena. [ref]
- 2023
S5 simplified the layer using a parallel scan for a multi-input system. [ref]
- 2024
A survey of state space models as an alternative to the Transformer. [ref]
Core idea
The system $x'=Ax+Bu$, $y=Cx+Du$ is discretized with step Δ and becomes a linear recurrence. Because it is linear and time-invariant, the recurrence can be unrolled into a convolution kernel $\bar K=(C\bar B, C\bar A\bar B, \dots)$ applied to the whole sequence at once (via FFT). Initializing A with HiPPO makes the state a compressed representation of history.
Interactive diagram
A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.
Select a layer in the diagram.
Components
| Component | Role |
|---|---|
| Continuous SSM (A, B, C, D) | Continuous linear model |
| Discretization | ZOH or bilinear with step Δ |
| HiPPO initialization | Initializing A to preserve history |
| Recurrent mode | For step-by-step inference |
| Convolutional mode | For parallel training |
| Nonlinearity + mixing | Between layers (GELU, linear) |
Equations
1. Continuous model
| $x(t)$ | State of dimension N |
| $u(t)$ | Input |
| $A,B,C,D$ | Parameters |
2. ZOH discretization
| $\Delta$ | Learned time step |
3. Convolution form
| $L$ | Sequence length |
How it works, step by step
- The system parameters are discretized with step Δ.
- The long convolution kernel is computed.
- The convolution is applied to the sequence via FFT.
- Du is added, then activation and channel mixing.
- At inference, the recurrence can be used at constant cost per step.
Strengths and limitations
Strengths
- Near-linear cost in length.
- Strong long-term memory on long-context benchmarks.
- Two modes: recurrent for inference and convolutional for training.
Limitations
- Linear time-invariant versions cannot choose what to remember based on content (addressed by Mamba).
- The full S4 implementation is mathematically complex.
When to use it, and when not?
Good for
- Very long sequences (raw audio, long series).
Not suited for
- Tasks that rely on exact copying and retrieval from context, where attention or selective SSMs win.
Applications
| Domain | Use |
|---|---|
| Time series forecasting | Forecasting long series. |
| Audio and speech | Raw audio modelling. |
| Natural language processing | Long-context language modelling. |
| Medical image analysis | Long ECG and EEG signals. |
PyTorch code
A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.
The original S4 uses a complex A matrix with HiPPO initialization and a low-rank representation; here is a simplified real diagonal version in the S4D style to illustrate discretization and convolution.
# State Space Model (simplified S4D): diagonal A matrix and convolution form
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
# 2) Define the model
class S4DLayer(nn.Module):
"""Real diagonal SSM per channel: ZOH discretization, then a long convolution kernel applied via FFT."""
def __init__(self, H, N=16):
super().__init__()
self.log_dt = nn.Parameter(torch.rand(H) * (torch.log(torch.tensor(0.1)) - torch.log(torch.tensor(0.001))) + torch.log(torch.tensor(0.001)))
self.log_A = nn.Parameter(torch.log(0.5 * torch.ones(H, N) + torch.arange(N))) # A = -exp(log_A) < 0 (stable)
self.B = nn.Parameter(torch.ones(H, N))
self.C = nn.Parameter(torch.randn(H, N) * 0.5)
self.D = nn.Parameter(torch.ones(H))
def kernel(self, L):
dt, A = self.log_dt.exp()[:, None], -self.log_A.exp() # H×1, H×N
Abar = torch.exp(dt * A)
Bbar = (Abar - 1) / A * self.B # ZOH for a diagonal matrix
powers = Abar[:, :, None] ** torch.arange(L) # H×N×L
return torch.einsum('hn,hnl->hl', self.C * Bbar, powers) # K̄ = C Ā^k B̄
def forward(self, u): # u: B×H×L
L = u.shape[-1]
K = self.kernel(L)
y = torch.fft.irfft(torch.fft.rfft(u, 2 * L) * torch.fft.rfft(K, 2 * L), 2 * L)[..., :L]
return y + self.D[:, None] * u
class SSMNet(nn.Module):
def __init__(self, H=16, n_classes=2):
super().__init__()
self.inp = nn.Linear(1, H)
self.layers = nn.ModuleList([S4DLayer(H), S4DLayer(H)])
self.mix = nn.ModuleList([nn.Linear(H, H), nn.Linear(H, H)])
self.out = nn.Linear(H, n_classes)
def forward(self, x): # B×T×1
h = self.inp(x)
for ssm, mix in zip(self.layers, self.mix):
h = h + mix(F.gelu(ssm(h.transpose(1, 2)).transpose(1, 2)))
return self.out(h.mean(1)) # mean pooling over time (as in S4 classification)
# 3) Prepare the data
# Memory task: the signal (±1) appears only at the first step, then noise; class = initial signal
def make(n, T=200):
y = torch.randint(0, 2, (n,))
X = torch.randn(n, T, 1) * 0.2
X[:, 0, 0] = y.float() * 2 - 1
return X, y
X_train, y_train = make(800)
X_test, y_test = make(200)
model = SSMNet()
# 4) Loss function and optimizer
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=5e-3)
# 5) Training loop with the forward pass
EPOCHS, BATCH = 15, 32
for epoch in range(EPOCHS):
model.train()
perm = torch.randperm(len(X_train))
total = 0.0
for i in range(0, len(X_train), BATCH):
idx = perm[i:i + BATCH]
xb, yb = X_train[idx], y_train[idx]
out = model(xb) # Forward Pass
loss = loss_fn(out, yb)
opt.zero_grad()
loss.backward() # Backpropagation
opt.step()
total += loss.item() * len(idx)
if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
print(f"epoch {epoch:3d} loss={total / len(X_train):.4f}")
# 6) Evaluation
model.eval()
with torch.no_grad():
xb = X_test
out = model(xb)
acc = (out.argmax(dim=-1) == y_test).float().mean().item()
print(f"test accuracy = {acc:.3f}")
# 7) Inference on a new sample
with torch.no_grad():
xb = X_test[:1]
out = model(xb)
print("predicted class:", out.argmax(dim=-1).item())
This code was run on 2026-11-20 in 5.8 seconds (PyTorch (CPU), Python 3 – local build machine).
Sample output:
test accuracy = 1.000
Trainable parameters in the example: model = 2,210.
These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.
Try it and go deeper
- Time Series Lab Another section of the site
Related architectures
Mamba: selective state space models
Mamba (Selective State Space Models)
An SSM in which B, C and the step Δ are functions of the input (selective), so the layer decides what to remember and what to ignore based on content, with a hardware-aw…
Temporal convolutional networks
Temporal Convolutional Network (TCN)
A causal 1D convolutional network (it cannot see the future) with dilated convolutions and residual connections, which processes sequences in parallel and covers a long…
Long short-term memory
Long Short-Term Memory (LSTM)
A recurrent network with a memory cell and gates (forget, input, output) that control what is kept and what is erased, so it learns long-term dependencies far better tha…
Transformer
Transformer
An architecture based entirely on multi-head self-attention instead of recurrence or convolution, so it processes all sequence positions in parallel and links any two po…
No results match the filters.
Research ideas
- Evaluating S4/S5 on long overnight EEG series against TCN and Transformer with an equal memory budget.
Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.
Test your understanding
Why can a linear time-invariant SSM layer be trained in parallel?
Why? Unrolling the recurrence gives $\bar K$.
With $A=-2$ and $\Delta=0.5$, what is $\bar A=e^{\Delta A}$?
Why? $0.5\times(-2)=-1$.
What is the role of HiPPO initialization in S4?
Why? HiPPO designs A to project history onto a polynomial basis.
References
Gu, A., Goel, K., & Ré, C. (2022). Efficiently modeling long sequences with structured state spaces. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2111.00396
Gu, A., Dao, T., Ermon, S., Rudra, A., & Ré, C. (2020). HiPPO: Recurrent memory with optimal polynomial projections. Advances in Neural Information Processing Systems (NeurIPS) 33. https://arxiv.org/abs/2008.07669
Smith, J. T. H., Warrington, A., & Linderman, S. W. (2023). Simplified state space layers for sequence modeling. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2208.04933
Wang, X., Wang, S., Ding, Y., Li, Y., Wu, W., Rong, Y., et al. (2026). State space model for new generation network alternative to transformers: A survey. Visual Intelligence, 4(1), Article 29 (2026; arXiv preprint 2404.09516, 2024). https://doi.org/10.1007/s44267-026-00133-1
