AdvancedGenerative NetworksHybrid Architectures

Latent Diffusion Models (LDM)

They apply diffusion in a compressed latent space from a pretrained autoencoder instead of pixel space, with a cross-attention mechanism for feeding in conditions (text, segmentation...), so computation drops sharply while quality is kept.

Overview

NameLatent Diffusion Models
Year introduced2022
ResearchersRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer
InstitutionLMU Munich / Heidelberg University / Runway
Original paperHigh-resolution image synthesis with latent diffusion models DOI arXiv
Data typeImages, Text
DomainsGenerative AI, Computer vision, Medical image analysis
Computational costFar less than pixel diffusion at the same resolution.
Data sizeVery large for general models; fine-tuning is possible with a few hundred images.

Problem it solves

Diffusion in pixel space is very costly for high-resolution images because each step processes every pixel, and most bits are imperceptible detail.

Background and evolution

What came before: Diffusion Models, Variational Autoencoder, Vector-Quantised Variational Autoencoder, Cross-attention, U-Net

  1. 2020

    DDPM in pixel space. [ref]

  2. 2022

    LDM (CVPR 2022), on which Stable Diffusion was built. [ref]

  3. 2023

    DiT replaced the U-Net with a transformer in latent diffusion. [ref]

Core idea

Two stages: (1) an autoencoder compresses the image by a factor $f$ (such as 8) into a latent $z$ with KL or VQ regularisation. (2) A diffusion model works on $z$. The condition $y$ is encoded by an encoder $\tau_\theta$ and enters the U-Net through cross-attention. To generate: reverse diffusion in the latent, then one decoder pass to pixels. Classifier-free guidance (CFG, Ho and Salimans) combines the conditional prediction with the "unconditional" one.

Example: A 512×512×3 image becomes a 64×64×4 latent (factor 8), about 48 times fewer values, so tens of diffusion steps are run on it.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Cross-attention: Q from the U-Net features, and K and V from τ(y).

Components

ComponentRole
Autoencoder (E, D)Perceptual compression
Latent U-Net ε_θDenoising in the latent
Condition encoder τ_θSuch as a text transformer
Cross-attention conditioning
Classifier-free guidanceAt generation time

Equations

1. Conditional LDM loss

$$\mathcal{L}_{LDM}=\mathbb{E}_{\mathcal{E}(x),y,\epsilon,t}\left\lVert\epsilon-\epsilon_\theta\big(z_t,t,\tau_\theta(y)\big)\right\rVert_2^2$$

Symbols
$z_t$Noisy latent
$\tau_\theta(y)$Representation of the condition
Dimensions
A scalar
Numerical example
Like DDPM, but on $z\in\mathbb{R}^{64\times64\times4}$ instead of $512\times512\times3$.
Relation to the architecture
The same diffusion in a smaller space.

2. Cross-attention for the condition

$$Q=W_Q\varphi_i(z_t),\quad K=W_K\tau_\theta(y),\quad V=W_V\tau_\theta(y)$$

Symbols
$\varphi_i$Intermediate U-Net features
Dimensions
The weights: (latent positions)$\times$(condition tokens)
Numerical example
A latent of $32\times32=1024$ positions and 77 text tokens ⇒ a $1024\times77$ matrix.
Relation to the architecture
It links each region to the appropriate words.

3. Classifier-free guidance (CFG)

$$\tilde\epsilon_\theta(z_t,c)=(1+w)\,\epsilon_\theta(z_t,c)-w\,\epsilon_\theta(z_t,\varnothing)$$

Symbols
$w$Guidance strength
$\varnothing$Empty condition
Dimensions
Same shape as ε
Numerical example
$w=3$, $\epsilon(c)=0.5$, $\epsilon(\varnothing)=0.3$: $4(0.5)-3(0.3)=1.1$.
Relation to the architecture
It strengthens adherence to the condition at the expense of diversity.

How it works, step by step

  1. Train the autoencoder.
  2. Encode the data into latents.
  3. Train diffusion on the latents, sometimes dropping the condition.
  4. Generate by reverse diffusion with CFG.
  5. Decode the latent into an image.

Strengths and limitations

Strengths

  • Large computational efficiency compared with pixel diffusion.
  • Flexible conditioning with cross-attention.
  • High quality at large resolutions.

Limitations

  • Quality limited by the autoencoder (very fine detail).
  • Risks of misuse and rights to the training data.

When to use it, and when not?

Good for

  • High-resolution conditional generation.
  • Editing and inpainting.

Not suited for

  • When full pixel accuracy requires guarantees the decoder does not provide.

Applications

DomainUse
Generative AIText-to-image generation and editing.
Computer visionSuper-resolution and inpainting.
Medical image analysisGenerating conditional medical images for research purposes.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

The condition here is a class (left/right half) through a simple embedding instead of cross-attention over text, to keep the example fast. The network is an MLP, not a U-Net.

Python (PyTorch)
# A miniature Latent Diffusion: autoencoder + conditional latent diffusion + CFG
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
class AE(nn.Module):                         # Stage 1: an autoencoder that compresses 8×8=64 pixels into a 4-dimensional latent
    def __init__(self, z=4):
        super().__init__()
        self.enc = nn.Sequential(nn.Flatten(), nn.Linear(64, 128), nn.ReLU(), nn.Linear(128, z))
        self.dec = nn.Sequential(nn.Linear(z, 128), nn.ReLU(), nn.Linear(128, 64), nn.Sigmoid())
    def forward(self, x):
        return self.dec(self.enc(x)).view_as(x)

class LatentEps(nn.Module):                  # Stage 2: a class-conditioned denoising network in the latent space
    def __init__(self, z=4, n_cond=3, T=100, h=128):
        super().__init__()
        self.T, self.cond = T, nn.Embedding(n_cond, 16)          # class 2 = "no condition" for CFG
        self.net = nn.Sequential(nn.Linear(z + 16 + 16, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, z))
    def forward(self, z, t, c):
        ang = (t.float() / self.T)[:, None] * torch.exp(torch.arange(8) * -1.0) * 50
        return self.net(torch.cat([z, ang.sin(), ang.cos(), self.cond(c)], 1))

# 3) Prepare the data
# 8×8 images of a Gaussian blob with a random position and radius (values in [0,1])
def blobs(n, side=None):
    g = torch.arange(8).float()
    cx = torch.rand(n) * 5 + 1.5
    if side is not None:                                   # side: 0 = left half, 1 = right half
        cx = torch.where(side == 0, torch.rand(n) * 2 + 1, torch.rand(n) * 2 + 5)
    cy, r = torch.rand(n) * 5 + 1.5, torch.rand(n) * 0.8 + 0.8
    img = torch.exp(-((g[None, None, :] - cx[:, None, None]) ** 2 + (g[None, :, None] - cy[:, None, None]) ** 2) / (2 * r[:, None, None] ** 2))
    return img[:, None]                                    # B×1×8×8

side_train = torch.randint(0, 2, (3000,)); X_train = blobs(3000, side_train)
side_test = torch.randint(0, 2, (500,));   X_test = blobs(500, side_test)
ae, model = AE(), LatentEps()
T = 100
betas = torch.linspace(1e-4, 0.05, T); alphas = 1 - betas; abar = torch.cumprod(alphas, 0)

# 4) Stage 1: train the autoencoder (reconstruction)
opt_ae = torch.optim.Adam(ae.parameters(), lr=2e-3)
for epoch in range(30):
    for i in range(0, len(X_train), 64):
        xb = X_train[i:i + 64]
        l = F.mse_loss(ae(xb), xb); opt_ae.zero_grad(); l.backward(); opt_ae.step()
with torch.no_grad():
    print(f"autoencoder test MSE = {F.mse_loss(ae(X_test), X_test).item():.4f}")
    Z = ae.enc(X_train); mu, sd = Z.mean(0), Z.std(0); Z = (Z - mu) / sd           # normalise the latent

# 5) Stage 2: DDPM diffusion in the latent space with the condition dropped at random (10%) to enable CFG
opt = torch.optim.Adam(model.parameters(), lr=2e-3)
for step in range(4000):
    idx = torch.randint(0, len(Z), (256,))
    z0, c = Z[idx], side_train[idx].clone()
    c[torch.rand(256) < 0.1] = 2
    t = torch.randint(0, T, (256,)); eps = torch.randn_like(z0)
    zt = abar[t, None].sqrt() * z0 + (1 - abar[t, None]).sqrt() * eps
    loss = F.mse_loss(model(zt, t, c), eps)                                  # Forward Pass
    opt.zero_grad(); loss.backward(); opt.step()
    if step % 1000 == 0 or step == 3999:
        print(f"step {step:4d}  diffusion loss={loss.item():.4f}")

# 6) Evaluation: does the blob appear in the requested half? (classifier-free guidance, w = 3)
@torch.no_grad()
def generate(c, w=3.0):
    n = len(c); z = torch.randn(n, 4)
    for t in reversed(range(T)):
        tt = torch.full((n,), t)
        e = (1 + w) * model(z, tt, c) - w * model(z, tt, torch.full((n,), 2))
        z = (z - betas[t] / (1 - abar[t]).sqrt() * e) / alphas[t].sqrt() + (betas[t].sqrt() * torch.randn_like(z) if t > 0 else 0)
    return ae.dec(z * sd + mu).view(n, 1, 8, 8)                              # decode to pixels
c = torch.randint(0, 2, (1000,))
img = generate(c)
col = (img[:, 0].sum(1) * torch.arange(8).float()).sum(1) / img[:, 0].sum((1, 2))   # horizontal centre of mass
print(f"conditional accuracy (blob on requested side) = {((col > 3.5).long() == c).float().mean().item():.3f}")

# 7) Inference: ask for a blob in the right half
im = generate(torch.tensor([1]))[0, 0]
print("requested right half → peak at (row, col):", divmod(im.argmax().item(), 8))

This code was run on 2026-11-20 in 10.7 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

autoencoder test MSE = 0.0004
conditional accuracy (blob on requested side) = 1.000

Trainable parameters in the example: ae = 17,732, model = 21,812.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Advanced2020

Diffusion Models

Diffusion Models (DDPM)

They gradually corrupt the data by adding Gaussian noise over T steps, then a network learns to reverse this process by predicting the added noise, so a new example is g…

Generative Networks

Intermediate2014

Variational Autoencoder

Variational Autoencoder (VAE)

A probabilistic generative model with a latent variable: an encoder that estimates an approximate distribution q(z x) and a decoder that represents p(x z), trained toget…

Generative Networks

Advanced2017

Vector-Quantised Variational Autoencoder

Vector Quantized Variational Autoencoder (VQ-VAE)

An autoencoder that quantises the encoder's output to the nearest vector in a learned codebook, so the representation becomes discrete codes; an autoregressive prior is…

Generative Networks

Basic2017

Cross-attention

Cross-Attention (Cross-Attn)

Attention in which the queries come from one sequence while the keys and values come from another sequence or modality, so each element of the first reads what it needs…

Transformer Architectures

Basic2015

U-Net

U-Net

A U-shaped segmentation architecture: a contracting path that captures context and an expanding path that restores spatial resolution, with skip connections that carry f…

FoundationsHybrid Architectures

Research2023

Flow Matching

Flow Matching (FM)

It learns the velocity field of an ordinary differential equation that carries the noise distribution to the data distribution, by regressing directly on the velocities…

Generative Networks

Research ideas

  • Fine-tune an LDM on medical images with differential privacy and measure the benefit for segmentation models.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    What is the core idea of LDM?

  2. Concept

    How does text enter the U-Net in LDM?

  3. Mathematical

    With $w=0$ in CFG, what does the prediction become?

References

  1. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 10674–10685. https://doi.org/10.1109/CVPR52688.2022.01042

    DOIarXivAutomatically verified via Crossref – 2026-11-20

  2. Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS) 33. https://arxiv.org/abs/2006.11239

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  3. Peebles, W., & Xie, S. (2023). Scalable diffusion models with transformers. IEEE/CVF International Conference on Computer Vision (ICCV), 4172–4182. https://doi.org/10.1109/ICCV51070.2023.00387

    DOIarXivAutomatically verified via Crossref – 2026-11-20

  4. Song, J., Meng, C., & Ermon, S. (2021). Denoising diffusion implicit models. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2010.02502

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  5. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., et al. (2021). Learning transferable visual models from natural language supervision. International Conference on Machine Learning (ICML), PMLR 139, 8748–8763. https://arxiv.org/abs/2103.00020

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20