Mamba: selective state space models (Mamba)
Mamba (Selective State Space Models)
An SSM in which B, C and the step Δ are functions of the input (selective), so the layer decides what to remember and what to ignore based on content, with a hardware-aware parallel scan algorithm that keeps cost linear in length.
Overview
| Name | Mamba: selective state space models – Mamba (Selective State Space Models) |
|---|---|
| Year introduced | 2023 |
| Researchers | Albert Gu, Tri Dao |
| Institution | Carnegie Mellon University / Princeton University |
| Original paper | Mamba: Linear-time sequence modeling with selective state spaces arXiv |
| Data type | Sequences and time series, Text |
| Domains | Time series forecasting, Audio and speech, Medical image analysis, Natural language processing |
| Computational cost | $O(L\cdot E\cdot N)$ for training and inference, with constant inference memory. |
| Data size | Usually large (language models), and also used on medium-size medical series. |
Problem it solves
Time-invariant SSMs (such as S4) cannot select information by content, so they are weak at tasks such as selective copying and retrieval, while attention has quadratic cost.
Background and evolution
What came before: State space models, Long short-term memory, Transformer
Core idea
In S4, $\bar A,\bar B,C$ are fixed for the whole sequence. In Mamba, $\Delta_t,B_t,C_t$ are computed from the input $x_t$ by linear projections, so the system becomes time-varying. A large Δ means "reset and focus on the current input", and a small Δ means "ignore the input and keep the state", which resembles the gates in LSTM. The loss of the convolution form is compensated by a parallel scan.
Interactive diagram
A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.
Select a layer in the diagram.
Components
| Component | Role |
|---|---|
| Input projection | Expanding the channel into two branches |
| Causal Conv1d | Short convolution before the SSM |
| Selective SSM (S6) | Δ, B and C depend on the input |
| Gating branch | SiLU as a multiplicative gate |
| Output projection | Restoring the dimensions |
| Hardware-aware scan | Parallel scan fused in SRAM |
Equations
1. Selective parameters
| $\Delta_t$ | A time step per token and channel |
| $B_t,C_t$ | Input-dependent projections |
2. Selective recurrence
| $A$ | Fixed negative diagonal |
| $h_t$ | State |
How it works, step by step
- The input is projected into two branches.
- A short causal convolution, then SiLU, is applied to the main branch.
- Δ, B and C are computed from the input.
- The selective scan is executed.
- The result is multiplied by the gate and projected back to the original dimensions with a residual connection.
Strengths and limitations
Strengths
- Linear cost in length and inference with a constant-size state.
- Content-based selection beyond fixed SSMs.
- Performance competitive with the Transformer in several domains, according to the authors.
Limitations
- The fast implementation needs custom CUDA kernels.
- Weaker on some exact in-context retrieval tasks than attention, according to later studies.
- A newer, less mature ecosystem.
When to use it, and when not?
Good for
- Very long sequences that need content selection.
- Memory-efficient language models.
Not suited for
- Very short tasks where efficiency brings no benefit.
- Environments without support for custom kernels, if speed is critical.
Applications
| Domain | Use |
|---|---|
| Natural language processing | Long-context language models. |
| Audio and speech | Audio modelling. |
| Medical image analysis | Mamba-based medical vision models (such as U-Mamba) and genomic sequences. |
| Time series forecasting | Forecasting long series. |
PyTorch code
A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.
A reference implementation with a time loop to illustrate selection; the official mamba-ssm library uses a parallel scan with CUDA kernels and different initializations.
# Mamba block (selective SSM) — slow reference implementation with a time loop
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
# 2) Define the model
class SelectiveSSM(nn.Module):
def __init__(self, E, N=8):
super().__init__()
self.A_log = nn.Parameter(torch.log(torch.arange(1, N + 1).float()).repeat(E, 1)) # A = -exp(A_log)
self.x_proj = nn.Linear(E, 1 + 2 * N) # → Δ (low-rank), B and C
self.dt_proj = nn.Linear(1, E)
nn.init.constant_(self.dt_proj.bias, -4.0) # small Δ initially (≈0.018) ⇒ longer memory
self.D, self.N = nn.Parameter(torch.ones(E)), N
def forward(self, x): # x: B×L×E
Bsz, L, E = x.shape
A = -self.A_log.exp() # E×N
dbc = self.x_proj(x)
dt = F.softplus(self.dt_proj(dbc[..., :1])) # B×L×E Δ_t
Bt, Ct = dbc[..., 1:1 + self.N], dbc[..., 1 + self.N:] # B×L×N
h, ys = torch.zeros(Bsz, E, self.N), []
for t in range(L): # the scan (sequential here for clarity)
dA = torch.exp(dt[:, t, :, None] * A) # exp(Δ_t A)
h = dA * h + dt[:, t, :, None] * Bt[:, t, None, :] * x[:, t, :, None]
ys.append((h * Ct[:, t, None, :]).sum(-1))
return torch.stack(ys, 1) + self.D * x
class MambaBlock(nn.Module):
def __init__(self, D=16, expand=2):
super().__init__()
E = expand * D
self.in_proj = nn.Linear(D, 2 * E)
self.conv = nn.Conv1d(E, E, 4, groups=E, padding=3) # short causal convolution
self.ssm, self.out_proj, self.norm = SelectiveSSM(E), nn.Linear(E, D), nn.LayerNorm(D)
def forward(self, x):
L = x.shape[1]
u, gate = self.in_proj(self.norm(x)).chunk(2, -1)
u = F.silu(self.conv(u.transpose(1, 2))[..., :L].transpose(1, 2))
return x + self.out_proj(self.ssm(u) * F.silu(gate)) # residual connection
class MambaClassifier(nn.Module):
def __init__(self, D=16, n_classes=2):
super().__init__()
self.inp, self.blocks, self.out = nn.Linear(1, D), nn.Sequential(MambaBlock(D), MambaBlock(D)), nn.Linear(D, n_classes)
def forward(self, x):
return self.out(self.blocks(self.inp(x))[:, -1])
# 3) Prepare the data
# Simplified selective-copy task: an important token (±1) at a random position amid noise; class = its sign
def make(n, T=60):
y = torch.randint(0, 2, (n,))
X = torch.randn(n, T, 1) * 0.2
pos = torch.randint(0, T // 2, (n,))
X[torch.arange(n), pos, 0] = (y.float() * 2 - 1) * 2
return X, y
X_train, y_train = make(600)
X_test, y_test = make(150)
model = MambaClassifier()
# 4) Loss function and optimizer
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=5e-3)
# 5) Training loop with the forward pass
EPOCHS, BATCH = 8, 32
for epoch in range(EPOCHS):
model.train()
perm = torch.randperm(len(X_train))
total = 0.0
for i in range(0, len(X_train), BATCH):
idx = perm[i:i + BATCH]
xb, yb = X_train[idx], y_train[idx]
out = model(xb) # Forward Pass
loss = loss_fn(out, yb)
opt.zero_grad()
loss.backward() # Backpropagation
opt.step()
total += loss.item() * len(idx)
if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
print(f"epoch {epoch:3d} loss={total / len(X_train):.4f}")
# 6) Evaluation
model.eval()
with torch.no_grad():
xb = X_test
out = model(xb)
acc = (out.argmax(dim=-1) == y_test).float().mean().item()
print(f"test accuracy = {acc:.3f}")
# 7) Inference on a new sample
with torch.no_grad():
xb = X_test[:1]
out = model(xb)
print("predicted class:", out.argmax(dim=-1).item())
This code was run on 2026-11-20 in 11.3 seconds (PyTorch (CPU), Python 3 – local build machine).
Sample output:
test accuracy = 1.000
Trainable parameters in the example: model = 5,508.
These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.
Try it and go deeper
Related architectures
State space models
Structured State Space Models (SSM / S4)
Sequence layers built on the continuous linear state space model, discretized in time and computable as a fast recurrence at inference or as a long parallel convolution…
Long short-term memory
Long Short-Term Memory (LSTM)
A recurrent network with a memory cell and gates (forget, input, output) that control what is kept and what is erased, so it learns long-term dependencies far better tha…
Transformer
Transformer
An architecture based entirely on multi-head self-attention instead of recurrence or convolution, so it processes all sequence positions in parallel and links any two po…
No results match the filters.
Research ideas
- Comparing Mamba and Transformers for 3D medical image segmentation within hybrid U-Net architectures.
- Studying Mamba's exact-retrieval ability in long Arabic texts.
Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.
Test your understanding
What makes Mamba "selective"?
Why? This makes the system time-varying according to content.
What happens when $\Delta_t$ is very small?
Why? $\exp(\Delta A)\approx1$ and $\Delta B x\approx0$.
Why can't Mamba use the convolution form as in S4?
Why? That is why it uses a parallel scan.
References
Gu, A., & Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint (later at COLM 2024). https://arxiv.org/abs/2312.00752
Dao, T., & Gu, A. (2024). Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. International Conference on Machine Learning (ICML), PMLR 235. https://arxiv.org/abs/2405.21060
Gu, A., Goel, K., & Ré, C. (2022). Efficiently modeling long sequences with structured state spaces. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2111.00396
