AdvancedTemporal and Sequential Networks

Mamba: selective state space models (Mamba)

Mamba (Selective State Space Models)

An SSM in which B, C and the step Δ are functions of the input (selective), so the layer decides what to remember and what to ignore based on content, with a hardware-aware parallel scan algorithm that keeps cost linear in length.

Overview

NameMamba: selective state space models – Mamba (Selective State Space Models)
Year introduced2023
ResearchersAlbert Gu, Tri Dao
InstitutionCarnegie Mellon University / Princeton University
Original paperMamba: Linear-time sequence modeling with selective state spaces arXiv
Data typeSequences and time series, Text
DomainsTime series forecasting, Audio and speech, Medical image analysis, Natural language processing
Computational cost$O(L\cdot E\cdot N)$ for training and inference, with constant inference memory.
Data sizeUsually large (language models), and also used on medium-size medical series.

Problem it solves

Time-invariant SSMs (such as S4) cannot select information by content, so they are weak at tasks such as selective copying and retrieval, while attention has quadratic cost.

Background and evolution

What came before: State space models, Long short-term memory, Transformer

  1. 2023

    Gu and Dao introduced Mamba with the selection mechanism and the hardware-aware parallel scan (arXiv, then COLM 2024). [ref]

  2. 2024

    Mamba-2 introduced the "structured state space duality" framework linking SSMs and attention. [ref]

Core idea

In S4, $\bar A,\bar B,C$ are fixed for the whole sequence. In Mamba, $\Delta_t,B_t,C_t$ are computed from the input $x_t$ by linear projections, so the system becomes time-varying. A large Δ means "reset and focus on the current input", and a small Δ means "ignore the input and keep the state", which resembles the gates in LSTM. The loss of the convolution form is compensated by a parallel scan.

Example: In a long sentence with noise (filler words), the layer learns to shrink Δ on filler words so the state is unaffected, and to enlarge it on important words.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
Input projectionExpanding the channel into two branches
Causal Conv1dShort convolution before the SSM
Selective SSM (S6)Δ, B and C depend on the input
Gating branchSiLU as a multiplicative gate
Output projectionRestoring the dimensions
Hardware-aware scanParallel scan fused in SRAM

Equations

1. Selective parameters

$$\Delta_t=\mathrm{softplus}\!\left(W_\Delta x_t+b_\Delta\right),\qquad B_t=W_Bx_t,\qquad C_t=W_Cx_t$$

Symbols
$\Delta_t$A time step per token and channel
$B_t,C_t$Input-dependent projections
Dimensions
$\Delta_t\in\mathbb{R}^{E}$, $B_t,C_t\in\mathbb{R}^{N}$
Numerical example
If the pre-softplus value is $-5$, then $\Delta\approx0.0067$ (keep the state); if it is $3$, then $\Delta\approx3.05$ (strong update).
Relation to the architecture
This is the "selection" that distinguishes Mamba.

2. Selective recurrence

$$h_t=\exp(\Delta_tA)\odot h_{t-1}+\Delta_tB_t\,x_t,\qquad y_t=C_t^{\top}h_t+Dx_t$$

Symbols
$A$Fixed negative diagonal
$h_t$State
Dimensions
$h_t\in\mathbb{R}^{E\times N}$
Numerical example
$A=-1$, $\Delta=0.01$ ⇒ $\exp(-0.01)\approx0.99$: the state stays almost unchanged.
Relation to the architecture
Uses the Euler approximation for $\bar B$, as in the reference implementation.

How it works, step by step

  1. The input is projected into two branches.
  2. A short causal convolution, then SiLU, is applied to the main branch.
  3. Δ, B and C are computed from the input.
  4. The selective scan is executed.
  5. The result is multiplied by the gate and projected back to the original dimensions with a residual connection.

Strengths and limitations

Strengths

  • Linear cost in length and inference with a constant-size state.
  • Content-based selection beyond fixed SSMs.
  • Performance competitive with the Transformer in several domains, according to the authors.

Limitations

  • The fast implementation needs custom CUDA kernels.
  • Weaker on some exact in-context retrieval tasks than attention, according to later studies.
  • A newer, less mature ecosystem.

When to use it, and when not?

Good for

  • Very long sequences that need content selection.
  • Memory-efficient language models.

Not suited for

  • Very short tasks where efficiency brings no benefit.
  • Environments without support for custom kernels, if speed is critical.

Applications

DomainUse
Natural language processingLong-context language models.
Audio and speechAudio modelling.
Medical image analysisMamba-based medical vision models (such as U-Mamba) and genomic sequences.
Time series forecastingForecasting long series.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

A reference implementation with a time loop to illustrate selection; the official mamba-ssm library uses a parallel scan with CUDA kernels and different initializations.

Python (PyTorch)
# Mamba block (selective SSM) — slow reference implementation with a time loop
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
class SelectiveSSM(nn.Module):
    def __init__(self, E, N=8):
        super().__init__()
        self.A_log = nn.Parameter(torch.log(torch.arange(1, N + 1).float()).repeat(E, 1))   # A = -exp(A_log)
        self.x_proj = nn.Linear(E, 1 + 2 * N)                        # → Δ (low-rank), B and C
        self.dt_proj = nn.Linear(1, E)
        nn.init.constant_(self.dt_proj.bias, -4.0)                   # small Δ initially (≈0.018) ⇒ longer memory
        self.D, self.N = nn.Parameter(torch.ones(E)), N
    def forward(self, x):                                            # x: B×L×E
        Bsz, L, E = x.shape
        A = -self.A_log.exp()                                        # E×N
        dbc = self.x_proj(x)
        dt = F.softplus(self.dt_proj(dbc[..., :1]))                  # B×L×E   Δ_t
        Bt, Ct = dbc[..., 1:1 + self.N], dbc[..., 1 + self.N:]       # B×L×N
        h, ys = torch.zeros(Bsz, E, self.N), []
        for t in range(L):                                           # the scan (sequential here for clarity)
            dA = torch.exp(dt[:, t, :, None] * A)                    # exp(Δ_t A)
            h = dA * h + dt[:, t, :, None] * Bt[:, t, None, :] * x[:, t, :, None]
            ys.append((h * Ct[:, t, None, :]).sum(-1))
        return torch.stack(ys, 1) + self.D * x

class MambaBlock(nn.Module):
    def __init__(self, D=16, expand=2):
        super().__init__()
        E = expand * D
        self.in_proj = nn.Linear(D, 2 * E)
        self.conv = nn.Conv1d(E, E, 4, groups=E, padding=3)          # short causal convolution
        self.ssm, self.out_proj, self.norm = SelectiveSSM(E), nn.Linear(E, D), nn.LayerNorm(D)
    def forward(self, x):
        L = x.shape[1]
        u, gate = self.in_proj(self.norm(x)).chunk(2, -1)
        u = F.silu(self.conv(u.transpose(1, 2))[..., :L].transpose(1, 2))
        return x + self.out_proj(self.ssm(u) * F.silu(gate))         # residual connection

class MambaClassifier(nn.Module):
    def __init__(self, D=16, n_classes=2):
        super().__init__()
        self.inp, self.blocks, self.out = nn.Linear(1, D), nn.Sequential(MambaBlock(D), MambaBlock(D)), nn.Linear(D, n_classes)
    def forward(self, x):
        return self.out(self.blocks(self.inp(x))[:, -1])

# 3) Prepare the data
# Simplified selective-copy task: an important token (±1) at a random position amid noise; class = its sign
def make(n, T=60):
    y = torch.randint(0, 2, (n,))
    X = torch.randn(n, T, 1) * 0.2
    pos = torch.randint(0, T // 2, (n,))
    X[torch.arange(n), pos, 0] = (y.float() * 2 - 1) * 2
    return X, y
X_train, y_train = make(600)
X_test, y_test = make(150)
model = MambaClassifier()

# 4) Loss function and optimizer
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=5e-3)

# 5) Training loop with the forward pass
EPOCHS, BATCH = 8, 32
for epoch in range(EPOCHS):
    model.train()
    perm = torch.randperm(len(X_train))
    total = 0.0
    for i in range(0, len(X_train), BATCH):
        idx = perm[i:i + BATCH]
        xb, yb = X_train[idx], y_train[idx]
        out = model(xb)                       # Forward Pass
        loss = loss_fn(out, yb)
        opt.zero_grad()
        loss.backward()                    # Backpropagation
        opt.step()
        total += loss.item() * len(idx)
    if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
        print(f"epoch {epoch:3d}  loss={total / len(X_train):.4f}")

# 6) Evaluation
model.eval()
with torch.no_grad():
    xb = X_test
    out = model(xb)
    acc = (out.argmax(dim=-1) == y_test).float().mean().item()
    print(f"test accuracy = {acc:.3f}")

# 7) Inference on a new sample
with torch.no_grad():
    xb = X_test[:1]
    out = model(xb)
    print("predicted class:", out.argmax(dim=-1).item())

This code was run on 2026-11-20 in 11.3 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

test accuracy = 1.000

Trainable parameters in the example: model = 5,508.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Advanced2022

State space models

Structured State Space Models (SSM / S4)

Sequence layers built on the continuous linear state space model, discretized in time and computable as a fast recurrence at inference or as a long parallel convolution…

Temporal and Sequential Networks

Basic1997

Long short-term memory

Long Short-Term Memory (LSTM)

A recurrent network with a memory cell and gates (forget, input, output) that control what is kept and what is erased, so it learns long-term dependencies far better tha…

Temporal and Sequential Networks

Basic2017

Transformer

Transformer

An architecture based entirely on multi-head self-attention instead of recurrence or convolution, so it processes all sequence positions in parallel and links any two po…

Transformer Architectures

Research ideas

  • Comparing Mamba and Transformers for 3D medical image segmentation within hybrid U-Net architectures.
  • Studying Mamba's exact-retrieval ability in long Arabic texts.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    What makes Mamba "selective"?

  2. Mathematical

    What happens when $\Delta_t$ is very small?

  3. Concept

    Why can't Mamba use the convolution form as in S4?

References

  1. Gu, A., & Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint (later at COLM 2024). https://arxiv.org/abs/2312.00752

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  2. Dao, T., & Gu, A. (2024). Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. International Conference on Machine Learning (ICML), PMLR 235. https://arxiv.org/abs/2405.21060

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  3. Gu, A., Goel, K., & Ré, C. (2022). Efficiently modeling long sequences with structured state spaces. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2111.00396

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20