IntermediateAdvanced Neural Architectures

Spiking Neural Networks (SNN)

Networks whose neurons communicate with discrete binary spikes in time, like the brain. The membrane potential accumulates and fires a spike when it exceeds the threshold, and is then reset.

Overview

NameSpiking Neural Networks
Year introduced1997
ResearchersWolfgang Maass (third-generation framing); training with surrogate gradients: Neftci, Mostafa, Zenke (2019)
Original paperNetworks of spiking neurons: The third generation of neural network models DOI
Data typeSequences and time series, Temporal events and spikes
DomainsComputer vision, Robotics, Audio and speech
Computational costOn a GPU: the cost of T time steps; on neuromorphic hardware: event-driven computation.
Data sizeMedium; often used with event data.

Problem it solves

Classic networks consume a lot of energy because they compute continuous values for every unit at every step. Neuromorphic computing aims at event-driven computation that consumes less.

Background and evolution

What came before: Multilayer Perceptron, Hodgkin–Huxley, Integrate-and-Fire

  1. 1997

    Maass classified spiking networks as a third generation and proved their computational power. [ref]

  2. 2014

    The reference book by Gerstner and colleagues on neuron dynamics. [ref]

  3. 2019

    Training with surrogate gradients became the common method. [ref]

Core idea

In the LIF model the potential leaks towards the resting value and rises with the input current. When it reaches the threshold $V_{th}$ a spike $S=1$ is fired and the potential is reset. The spike function (a Heaviside step) is not differentiable, so its gradient is replaced in backpropagation by a smooth function (a surrogate).

Example: A neuron receiving a constant current: if the current is weak, the potential never reaches the threshold; as the current increases, the firing rate increases. Information is encoded in the spike rate or timing.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
Input encodingRate (Poisson) or temporal encoding, or event data
LIF neuronsA membrane potential that leaks and accumulates
Threshold & resetFiring a spike and resetting
Synaptic weightsWeights between layers
Surrogate gradientApproximating the spike derivative
ReadoutSpike counts or the output layer potential

Equations

1. Continuous LIF equation

$$\tau_m\frac{dV}{dt}=-(V-V_{rest})+R\,I(t)$$

Symbols
$\tau_m$Membrane time constant
$V_{rest}$Resting potential
$R$Resistance
$I(t)$Input current
Dimensions
Scalar quantities for each neuron
Numerical example
With $V_{rest}=0$ and a constant $RI=1.5$, the potential approaches $1.5$; if the threshold is $1$, the neuron fires periodic spikes.
Relation to the architecture
It describes accumulation and leak.

2. The discrete form with firing

$$V_t=\beta V_{t-1}+W x_t-S_{t-1}V_{th},\qquad S_t=\Theta(V_t-V_{th}),\qquad \beta=e^{-\Delta t/\tau_m}$$

Symbols
$\beta$Leak factor
$\Theta$Step function
$S_t$The spike is 0 or 1
Dimensions
For each neuron at each step
Numerical example
$\beta=0.9$, $V_{t-1}=0.8$, $Wx_t=0.3$ ⇒ $V_t=1.02>1$, so a spike is fired.
Relation to the architecture
This is what the simulator in the lab and the code implement.

3. Surrogate gradient

$$\frac{\partial S}{\partial V}\approx\frac{1}{\left(1+k\,|V-V_{th}|\right)^2}$$

Symbols
$k$Steepness of the curve
Dimensions
A scalar
Numerical example
At $V=V_{th}$ the gradient is 1, and it decreases away from the threshold.
Relation to the architecture
It allows training with backpropagation through time.

How it works, step by step

  1. The input is encoded as spike trains over T steps.
  2. At each step the current is computed and the potential updated.
  3. Spikes are fired and the potential reset.
  4. The output spikes are counted.
  5. The gradient through time is computed with the smooth surrogate.

Strengths and limitations

Strengths

  • Potential energy efficiency on neuromorphic hardware.
  • Well suited to event data (event cameras).
  • Natural temporal modelling.

Limitations

  • Training is harder and slower on ordinary GPUs.
  • Accuracy is often lower than equivalent classic networks.
  • Energy gains are realised on specialised hardware.

When to use it, and when not?

Good for

  • Low-power embedded systems.
  • Event sensor data.

Not suited for

  • When top accuracy on a conventional GPU is the priority.

Applications

DomainUse
RoboticsLow-power control in robotics.
Audio and speechKeyword spotting.
Computer visionProcessing event-camera data.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

Libraries such as snnTorch or SpikingJelly can be used for larger models; here is a hand-written implementation to illustrate the equations.

Python (PyTorch)
# Spiking Neural Network with LIF neurons and a surrogate gradient
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
class SpikeFn(torch.autograd.Function):
    @staticmethod
    def forward(ctx, v):
        ctx.save_for_backward(v)
        return (v > 0).float()                              # Θ(V - V_th) after the shift
    @staticmethod
    def backward(ctx, g):
        v, = ctx.saved_tensors
        return g / (1 + 10 * v.abs()) ** 2                  # surrogate gradient
spike = SpikeFn.apply

class SNN(nn.Module):
    def __init__(self, d_in=20, h=64, n_classes=2, beta=0.9, T=25):
        super().__init__()
        self.fc1, self.fc2, self.beta, self.T = nn.Linear(d_in, h), nn.Linear(h, n_classes), beta, T
    def forward(self, x):                                   # x: B×d in [0,1] ⇒ rate encoding
        B = x.shape[0]
        v1, v2 = torch.zeros(B, self.fc1.out_features), torch.zeros(B, self.fc2.out_features)
        count = torch.zeros(B, self.fc2.out_features)
        for t in range(self.T):
            s_in = (torch.rand_like(x) < x).float()         # Poisson spikes
            v1 = self.beta * v1 + self.fc1(s_in); s1 = spike(v1 - 1.0); v1 = v1 - s1   # reset by subtraction
            v2 = self.beta * v2 + self.fc2(s1);   s2 = spike(v2 - 1.0); v2 = v2 - s2
            count = count + s2
        return count                                        # spike count of each class

# 3) Prepare the data
# Two classes: a high-intensity pattern in the first half of the inputs or in the second half
y = torch.randint(0, 2, (600,))
X = torch.rand(600, 20) * 0.3
X[y == 0, :10] += 0.6; X[y == 1, 10:] += 0.6
X = X.clamp(0, 1)
X_train, y_train, X_test, y_test = X[:480], y[:480], X[480:], y[480:]
model = SNN()

# 4) Loss function and optimizer
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=5e-3)

# 5) Training loop with the forward pass
EPOCHS, BATCH = 8, 32
for epoch in range(EPOCHS):
    model.train()
    perm = torch.randperm(len(X_train))
    total = 0.0
    for i in range(0, len(X_train), BATCH):
        idx = perm[i:i + BATCH]
        xb, yb = X_train[idx], y_train[idx]
        out = model(xb)                       # Forward Pass
        loss = loss_fn(out, yb)
        opt.zero_grad()
        loss.backward()                    # Backpropagation
        opt.step()
        total += loss.item() * len(idx)
    if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
        print(f"epoch {epoch:3d}  loss={total / len(X_train):.4f}")

# 6) Evaluation
model.eval()
with torch.no_grad():
    xb = X_test
    out = model(xb)
    acc = (out.argmax(dim=-1) == y_test).float().mean().item()
    print(f"test accuracy = {acc:.3f}")

# 7) Inference on a new sample
with torch.no_grad():
    xb = X_test[:1]
    out = model(xb)
    print("predicted class:", out.argmax(dim=-1).item())

This code was run on 2026-11-20 in 3.5 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

test accuracy = 1.000

Trainable parameters in the example: model = 1,474.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Advanced2021

Liquid Neural Networks

Liquid Neural Networks (LNN)

Continuous-time recurrent networks in which each neuron's time constant changes with the input (Liquid Time-Constant), so their dynamics adapt to the data with a small n…

Temporal and Sequential NetworksAdvanced Neural Architectures

Basic2004

Reservoir Computing

Reservoir Computing (ESN / LSM)

A large recurrent network with fixed random weights (the reservoir) turns the input into rich states, and only the linear readout layer is trained, usually with closed-f…

Temporal and Sequential NetworksAdvanced Neural Architectures

Beginner1990

Recurrent neural network

Recurrent Neural Network (RNN)

A network that processes a sequence element by element and carries a hidden state that passes information from previous steps, using the same weights at every step.

Temporal and Sequential Networks

Research ideas

  • Estimate the actual energy consumption of an SNN on neuromorphic hardware versus a quantised CNN for an Arabic keyword-spotting task.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    When does an LIF neuron fire a spike?

  2. Concept

    Why do we need a surrogate gradient?

  3. Mathematical

    With $\beta=0.5$, $V_{t-1}=1.2$, an input of $0.5$ and a threshold of 1, does the neuron fire?

References

  1. Maass, W. (1997). Networks of spiking neurons: The third generation of neural network models. Neural Networks, 10(9), 1659–1671. https://doi.org/10.1016/S0893-6080(97)00011-7

    DOIAutomatically verified via Crossref – 2026-11-20

  2. Neftci, E. O., Mostafa, H., & Zenke, F. (2019). Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6), 51–63. https://doi.org/10.1109/MSP.2019.2931595

    DOIarXivAutomatically verified via Crossref – 2026-11-20

  3. Gerstner, W., Kistler, W. M., Naud, R., & Paninski, L. (2014). Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press. https://doi.org/10.1017/CBO9781107447615

    DOIAutomatically verified via Crossref – 2026-11-20