ResearchGenerative Networks

Score-Based Generative Models (SGM)

They learn a "score" network $\nabla_x\log p(x)$ at multiple noise levels, then generate samples with Langevin dynamics that follow the score from noise towards high-density regions; they generalise to stochastic differential equations (SDEs) that unify them with diffusion.

Overview

NameScore-Based Generative Models
Year introduced2019
ResearchersYang Song, Stefano Ermon
InstitutionStanford University
Original paperGenerative modeling by estimating gradients of the data distribution arXiv
Data typeImages, Tabular data
DomainsScientific computing, Generative AI, Medical image analysis
Computational costMany network evaluations during generation.
Data sizeMedium to large.

Problem it solves

Learning the density directly requires an intractable normalising constant, which the score does not need. But estimating the score in data-sparse regions is inaccurate, which spoils sampling.

Background and evolution

What came before: Energy-based models, Autoencoder

  1. 2011

    Vincent established the link between denoising autoencoders and score matching. [ref]

  2. 2019

    NCSN: a noise-conditioned score + annealed Langevin dynamics. [ref]

  3. 2021

    Score SDE: a unified framework for diffusion and score models with a probability-flow ODE. [ref]

Core idea

Add noise at levels $\sigma_1>\dots>\sigma_L$ to cover the empty space. Train $s_\theta(x,\sigma)$ with denoising score matching: the target score for Gaussian noise is known, $-(\tilde x-x)/\sigma^2$. To generate, start with large noise and apply Langevin steps at each level in decreasing order. In the continuous limit this becomes a reverse SDE.

Example: Sampling crystal or molecular structures: the score network pushes atoms towards stable, high-probability configurations.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
Noise levels σ_iGeometric schedule
Noise-conditional score networks_θ(x, σ)
Denoising score matching
Annealed Langevin / reverse SDESampling
Probability flow ODEDeterministic alternative

Equations

1. Denoising score matching

$$\mathcal{L}(\theta;\sigma)=\tfrac12\,\mathbb{E}_{x,\tilde x}\left\lVert s_\theta(\tilde x,\sigma)+\frac{\tilde x-x}{\sigma^2}\right\rVert^2$$

Symbols
$\tilde x$$x+\sigma z$
Dimensions
A scalar
Numerical example
$x=1$, $\sigma=0.5$, $\tilde x=1.25$ ⇒ the target score is $-(0.25)/0.25=-1$.
Relation to the architecture
A tractable objective equivalent to matching the score of the noisy distribution.

2. Langevin dynamics

$$x_{k+1}=x_k+\frac{\alpha}{2}\,s_\theta(x_k,\sigma)+\sqrt{\alpha}\,z_k,\qquad z_k\sim\mathcal N(0,I)$$

Symbols
$\alpha$Step size
Dimensions
Same shape as x
Numerical example
$x=0$, $s=2$, $\alpha=0.01$, $z=0$ ⇒ $x=0.01$.
Relation to the architecture
It climbs the log-density hill, with noise that ensures exploration.

3. The reverse SDE

$$dx=\left[f(x,t)-g(t)^2\nabla_x\log p_t(x)\right]dt+g(t)\,d\bar w$$

Symbols
$f,g$The two coefficients of the forward SDE
$\bar w$Reverse-time Brownian motion
Dimensions
A path in data space
Numerical example
In the VE version: $f=0$ and $g=\sqrt{d\sigma^2/dt}$.
Relation to the architecture
It includes NCSN and DDPM as two special cases.

How it works, step by step

  1. Choose the noise levels.
  2. Train the score network with DSM.
  3. Start from large noise.
  4. Apply Langevin steps for each σ in decreasing order.
  5. Or solve the reverse SDE/ODE.

Strengths and limitations

Strengths

  • No need for the normalising constant.
  • A unified theoretical framework with diffusion.
  • Exact likelihoods through the ODE.

Limitations

  • Slow sampling.
  • Sensitivity to the noise schedule and the step size.

When to use it, and when not?

Good for

  • Modelling complex distributions.
  • Inverse problems and science.

Not suited for

  • When real-time generation is required.

Applications

DomainUse
Generative AIImage generation.
Scientific computingGenerating molecules and materials.
Medical image analysisReconstructing images from incomplete measurements.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

Evaluation here uses two simple metrics computed inside the example: the share of samples close to one of the eight modes, and the number of modes covered. FID is not used because it is specific to images.

Python (PyTorch)
# NCSN — denoising score matching and annealed Langevin dynamics
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
class ScoreNet(nn.Module):
    """s_θ(x, σ) ≈ ∇_x log p_σ(x), conditioned on the noise level (NCSN)."""
    def __init__(self, h=128):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(3, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 2))
    def forward(self, x, sigma):
        return self.net(torch.cat([x, sigma.log()[:, None]], 1)) / sigma[:, None]   # dividing by σ stabilises the scale

SIGMAS = torch.exp(torch.linspace(torch.log(torch.tensor(3.0)), torch.log(torch.tensor(0.05)), 10))   # geometric noise levels
model = ScoreNet()

# 3) Prepare the data
# Target distribution: a mixture of 8 Gaussians on a circle of radius 2 (a classic test of mode collapse)
CENTERS = torch.stack([torch.tensor([2 * torch.cos(torch.tensor(k * torch.pi / 4)), 2 * torch.sin(torch.tensor(k * torch.pi / 4))]) for k in range(8)])
def sample_ring(n, std=0.1):
    k = torch.randint(0, 8, (n,))
    return CENTERS[k] + std * torch.randn(n, 2), k

def ring_metrics(x):
    """Quality: share of samples close (<0.3) to one of the centres. Coverage: number of modes that received ≥5% of the good samples."""
    d = torch.cdist(x, CENTERS)
    near, k = d.min(1)
    good = near < 0.3
    counts = torch.bincount(k[good], minlength=8).float()
    covered = int((counts >= 0.05 * max(1, good.sum().item())).sum())
    return good.float().mean().item(), covered

# 4) Denoising score matching (DSM) with weight λ(σ)=σ²:  || σ s_θ(x̃,σ) + (x̃−x)/σ ||²
opt = torch.optim.Adam(model.parameters(), lr=2e-3)

# 5) Training loop
STEPS, BATCH = 4000, 256
for step in range(STEPS):
    x, _ = sample_ring(BATCH)
    sigma = SIGMAS[torch.randint(0, len(SIGMAS), (BATCH,))]
    noise = torch.randn_like(x) * sigma[:, None]
    s = model(x + noise, sigma)                                          # Forward Pass
    loss = ((sigma[:, None] * s + noise / sigma[:, None]) ** 2).sum(1).mean() / 2
    opt.zero_grad(); loss.backward(); opt.step()
    if step % 1000 == 0 or step == STEPS - 1:
        print(f"step {step:4d}  loss={loss.item():.4f}")

# 6) Evaluation: annealed Langevin dynamics
@torch.no_grad()
def sample(n, steps_per_sigma=50, eps=2e-4):
    x = torch.randn(n, 2) * SIGMAS[0]
    for sigma in SIGMAS:
        alpha = eps * (sigma / SIGMAS[-1]) ** 2                          # step size proportional to σ²
        for _ in range(steps_per_sigma):
            x = x + alpha / 2 * model(x, sigma.expand(n)) + alpha.sqrt() * torch.randn_like(x)
    return x
q, cov = ring_metrics(sample(4000))
print(f"sample quality (near a mode) = {q:.3f} ; modes covered = {cov}/8")

# 7) Inference
print("generated point:", [round(v, 3) for v in sample(1)[0].tolist()])

This code was run on 2026-11-20 in 10.6 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

sample quality (near a mode) = 0.918 ; modes covered = 8/8

Trainable parameters in the example: model = 33,794.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Advanced2020

Diffusion Models

Diffusion Models (DDPM)

They gradually corrupt the data by adding Gaussian noise over T steps, then a network learns to reverse this process by predicting the added noise, so a new example is g…

Generative Networks

Advanced2006

Energy-based models

Energy-Based Models (EBM)

Models that learn a scalar energy function E_θ(x) that is low for likely configurations and high for others, and define the density implicitly p(x) e^-E(x) without compu…

Generative NetworksSpecialized Architectures

Research2023

Flow Matching

Flow Matching (FM)

It learns the velocity field of an ordinary differential equation that carries the noise distribution to the data distribution, by regressing directly on the velocities…

Generative Networks

Research ideas

  • Score models for low-dose CT reconstruction with guarantees of consistency with the measurements.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    Why are multiple noise levels used?

  2. Mathematical

    What is the target score for $\tilde x=x+\sigma z$?

  3. Concept

    What did the Score SDE framework unify?

References

  1. Song, Y., & Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems (NeurIPS) 32. https://arxiv.org/abs/1907.05600

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  2. Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2021). Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2011.13456

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  3. Vincent, P. (2011). A connection between score matching and denoising autoencoders. Neural Computation, 23(7), 1661–1674. https://doi.org/10.1162/NECO_a_00142

    DOIAutomatically verified via Crossref – 2026-11-20

  4. Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS) 33. https://arxiv.org/abs/2006.11239

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20