Score-Based Generative Models (SGM)
They learn a "score" network $\nabla_x\log p(x)$ at multiple noise levels, then generate samples with Langevin dynamics that follow the score from noise towards high-density regions; they generalise to stochastic differential equations (SDEs) that unify them with diffusion.
Overview
| Name | Score-Based Generative Models |
|---|---|
| Year introduced | 2019 |
| Researchers | Yang Song, Stefano Ermon |
| Institution | Stanford University |
| Original paper | Generative modeling by estimating gradients of the data distribution arXiv |
| Data type | Images, Tabular data |
| Domains | Scientific computing, Generative AI, Medical image analysis |
| Computational cost | Many network evaluations during generation. |
| Data size | Medium to large. |
Problem it solves
Learning the density directly requires an intractable normalising constant, which the score does not need. But estimating the score in data-sparse regions is inaccurate, which spoils sampling.
Background and evolution
What came before: Energy-based models, Autoencoder
Core idea
Add noise at levels $\sigma_1>\dots>\sigma_L$ to cover the empty space. Train $s_\theta(x,\sigma)$ with denoising score matching: the target score for Gaussian noise is known, $-(\tilde x-x)/\sigma^2$. To generate, start with large noise and apply Langevin steps at each level in decreasing order. In the continuous limit this becomes a reverse SDE.
Interactive diagram
A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.
Select a layer in the diagram.
Components
| Component | Role |
|---|---|
| Noise levels σ_i | Geometric schedule |
| Noise-conditional score network | s_θ(x, σ) |
| Denoising score matching | |
| Annealed Langevin / reverse SDE | Sampling |
| Probability flow ODE | Deterministic alternative |
Equations
1. Denoising score matching
| $\tilde x$ | $x+\sigma z$ |
2. Langevin dynamics
| $\alpha$ | Step size |
3. The reverse SDE
| $f,g$ | The two coefficients of the forward SDE |
| $\bar w$ | Reverse-time Brownian motion |
How it works, step by step
- Choose the noise levels.
- Train the score network with DSM.
- Start from large noise.
- Apply Langevin steps for each σ in decreasing order.
- Or solve the reverse SDE/ODE.
Strengths and limitations
Strengths
- No need for the normalising constant.
- A unified theoretical framework with diffusion.
- Exact likelihoods through the ODE.
Limitations
- Slow sampling.
- Sensitivity to the noise schedule and the step size.
When to use it, and when not?
Good for
- Modelling complex distributions.
- Inverse problems and science.
Not suited for
- When real-time generation is required.
Applications
| Domain | Use |
|---|---|
| Generative AI | Image generation. |
| Scientific computing | Generating molecules and materials. |
| Medical image analysis | Reconstructing images from incomplete measurements. |
PyTorch code
A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.
Evaluation here uses two simple metrics computed inside the example: the share of samples close to one of the eight modes, and the number of modes covered. FID is not used because it is specific to images.
# NCSN — denoising score matching and annealed Langevin dynamics
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
# 2) Define the model
class ScoreNet(nn.Module):
"""s_θ(x, σ) ≈ ∇_x log p_σ(x), conditioned on the noise level (NCSN)."""
def __init__(self, h=128):
super().__init__()
self.net = nn.Sequential(nn.Linear(3, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 2))
def forward(self, x, sigma):
return self.net(torch.cat([x, sigma.log()[:, None]], 1)) / sigma[:, None] # dividing by σ stabilises the scale
SIGMAS = torch.exp(torch.linspace(torch.log(torch.tensor(3.0)), torch.log(torch.tensor(0.05)), 10)) # geometric noise levels
model = ScoreNet()
# 3) Prepare the data
# Target distribution: a mixture of 8 Gaussians on a circle of radius 2 (a classic test of mode collapse)
CENTERS = torch.stack([torch.tensor([2 * torch.cos(torch.tensor(k * torch.pi / 4)), 2 * torch.sin(torch.tensor(k * torch.pi / 4))]) for k in range(8)])
def sample_ring(n, std=0.1):
k = torch.randint(0, 8, (n,))
return CENTERS[k] + std * torch.randn(n, 2), k
def ring_metrics(x):
"""Quality: share of samples close (<0.3) to one of the centres. Coverage: number of modes that received ≥5% of the good samples."""
d = torch.cdist(x, CENTERS)
near, k = d.min(1)
good = near < 0.3
counts = torch.bincount(k[good], minlength=8).float()
covered = int((counts >= 0.05 * max(1, good.sum().item())).sum())
return good.float().mean().item(), covered
# 4) Denoising score matching (DSM) with weight λ(σ)=σ²: || σ s_θ(x̃,σ) + (x̃−x)/σ ||²
opt = torch.optim.Adam(model.parameters(), lr=2e-3)
# 5) Training loop
STEPS, BATCH = 4000, 256
for step in range(STEPS):
x, _ = sample_ring(BATCH)
sigma = SIGMAS[torch.randint(0, len(SIGMAS), (BATCH,))]
noise = torch.randn_like(x) * sigma[:, None]
s = model(x + noise, sigma) # Forward Pass
loss = ((sigma[:, None] * s + noise / sigma[:, None]) ** 2).sum(1).mean() / 2
opt.zero_grad(); loss.backward(); opt.step()
if step % 1000 == 0 or step == STEPS - 1:
print(f"step {step:4d} loss={loss.item():.4f}")
# 6) Evaluation: annealed Langevin dynamics
@torch.no_grad()
def sample(n, steps_per_sigma=50, eps=2e-4):
x = torch.randn(n, 2) * SIGMAS[0]
for sigma in SIGMAS:
alpha = eps * (sigma / SIGMAS[-1]) ** 2 # step size proportional to σ²
for _ in range(steps_per_sigma):
x = x + alpha / 2 * model(x, sigma.expand(n)) + alpha.sqrt() * torch.randn_like(x)
return x
q, cov = ring_metrics(sample(4000))
print(f"sample quality (near a mode) = {q:.3f} ; modes covered = {cov}/8")
# 7) Inference
print("generated point:", [round(v, 3) for v in sample(1)[0].tolist()])
This code was run on 2026-11-20 in 10.6 seconds (PyTorch (CPU), Python 3 – local build machine).
Sample output:
sample quality (near a mode) = 0.918 ; modes covered = 8/8
Trainable parameters in the example: model = 33,794.
These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.
Try it and go deeper
Related architectures
Diffusion Models
Diffusion Models (DDPM)
They gradually corrupt the data by adding Gaussian noise over T steps, then a network learns to reverse this process by predicting the added noise, so a new example is g…
Energy-based models
Energy-Based Models (EBM)
Models that learn a scalar energy function E_θ(x) that is low for likely configurations and high for others, and define the density implicitly p(x) e^-E(x) without compu…
Flow Matching
Flow Matching (FM)
It learns the velocity field of an ordinary differential equation that carries the noise distribution to the data distribution, by regressing directly on the velocities…
No results match the filters.
Research ideas
- Score models for low-dose CT reconstruction with guarantees of consistency with the measurements.
Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.
Test your understanding
Why are multiple noise levels used?
Why? The level is then lowered gradually.
What is the target score for $\tilde x=x+\sigma z$?
Why? The gradient of the log of the Gaussian density.
What did the Score SDE framework unify?
Why? As two special cases of an SDE.
References
Song, Y., & Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems (NeurIPS) 32. https://arxiv.org/abs/1907.05600
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2021). Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2011.13456
Vincent, P. (2011). A connection between score matching and denoising autoencoders. Neural Computation, 23(7), 1661–1674. https://doi.org/10.1162/NECO_a_00142
Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS) 33. https://arxiv.org/abs/2006.11239
