Latent Diffusion Models (LDM)
They apply diffusion in a compressed latent space from a pretrained autoencoder instead of pixel space, with a cross-attention mechanism for feeding in conditions (text, segmentation...), so computation drops sharply while quality is kept.
Overview
| Name | Latent Diffusion Models |
|---|---|
| Year introduced | 2022 |
| Researchers | Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer |
| Institution | LMU Munich / Heidelberg University / Runway |
| Original paper | High-resolution image synthesis with latent diffusion models DOI arXiv |
| Data type | Images, Text |
| Domains | Generative AI, Computer vision, Medical image analysis |
| Computational cost | Far less than pixel diffusion at the same resolution. |
| Data size | Very large for general models; fine-tuning is possible with a few hundred images. |
Problem it solves
Diffusion in pixel space is very costly for high-resolution images because each step processes every pixel, and most bits are imperceptible detail.
Background and evolution
What came before: Diffusion Models, Variational Autoencoder, Vector-Quantised Variational Autoencoder, Cross-attention, U-Net
Core idea
Two stages: (1) an autoencoder compresses the image by a factor $f$ (such as 8) into a latent $z$ with KL or VQ regularisation. (2) A diffusion model works on $z$. The condition $y$ is encoded by an encoder $\tau_\theta$ and enters the U-Net through cross-attention. To generate: reverse diffusion in the latent, then one decoder pass to pixels. Classifier-free guidance (CFG, Ho and Salimans) combines the conditional prediction with the "unconditional" one.
Interactive diagram
A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.
Select a layer in the diagram.
Cross-attention: Q from the U-Net features, and K and V from τ(y).
Components
| Component | Role |
|---|---|
| Autoencoder (E, D) | Perceptual compression |
| Latent U-Net ε_θ | Denoising in the latent |
| Condition encoder τ_θ | Such as a text transformer |
| Cross-attention conditioning | |
| Classifier-free guidance | At generation time |
Equations
1. Conditional LDM loss
| $z_t$ | Noisy latent |
| $\tau_\theta(y)$ | Representation of the condition |
2. Cross-attention for the condition
| $\varphi_i$ | Intermediate U-Net features |
3. Classifier-free guidance (CFG)
| $w$ | Guidance strength |
| $\varnothing$ | Empty condition |
How it works, step by step
- Train the autoencoder.
- Encode the data into latents.
- Train diffusion on the latents, sometimes dropping the condition.
- Generate by reverse diffusion with CFG.
- Decode the latent into an image.
Strengths and limitations
Strengths
- Large computational efficiency compared with pixel diffusion.
- Flexible conditioning with cross-attention.
- High quality at large resolutions.
Limitations
- Quality limited by the autoencoder (very fine detail).
- Risks of misuse and rights to the training data.
When to use it, and when not?
Good for
- High-resolution conditional generation.
- Editing and inpainting.
Not suited for
- When full pixel accuracy requires guarantees the decoder does not provide.
Applications
| Domain | Use |
|---|---|
| Generative AI | Text-to-image generation and editing. |
| Computer vision | Super-resolution and inpainting. |
| Medical image analysis | Generating conditional medical images for research purposes. |
PyTorch code
A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.
The condition here is a class (left/right half) through a simple embedding instead of cross-attention over text, to keep the example fast. The network is an MLP, not a U-Net.
# A miniature Latent Diffusion: autoencoder + conditional latent diffusion + CFG
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
# 2) Define the model
class AE(nn.Module): # Stage 1: an autoencoder that compresses 8×8=64 pixels into a 4-dimensional latent
def __init__(self, z=4):
super().__init__()
self.enc = nn.Sequential(nn.Flatten(), nn.Linear(64, 128), nn.ReLU(), nn.Linear(128, z))
self.dec = nn.Sequential(nn.Linear(z, 128), nn.ReLU(), nn.Linear(128, 64), nn.Sigmoid())
def forward(self, x):
return self.dec(self.enc(x)).view_as(x)
class LatentEps(nn.Module): # Stage 2: a class-conditioned denoising network in the latent space
def __init__(self, z=4, n_cond=3, T=100, h=128):
super().__init__()
self.T, self.cond = T, nn.Embedding(n_cond, 16) # class 2 = "no condition" for CFG
self.net = nn.Sequential(nn.Linear(z + 16 + 16, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(), nn.Linear(h, z))
def forward(self, z, t, c):
ang = (t.float() / self.T)[:, None] * torch.exp(torch.arange(8) * -1.0) * 50
return self.net(torch.cat([z, ang.sin(), ang.cos(), self.cond(c)], 1))
# 3) Prepare the data
# 8×8 images of a Gaussian blob with a random position and radius (values in [0,1])
def blobs(n, side=None):
g = torch.arange(8).float()
cx = torch.rand(n) * 5 + 1.5
if side is not None: # side: 0 = left half, 1 = right half
cx = torch.where(side == 0, torch.rand(n) * 2 + 1, torch.rand(n) * 2 + 5)
cy, r = torch.rand(n) * 5 + 1.5, torch.rand(n) * 0.8 + 0.8
img = torch.exp(-((g[None, None, :] - cx[:, None, None]) ** 2 + (g[None, :, None] - cy[:, None, None]) ** 2) / (2 * r[:, None, None] ** 2))
return img[:, None] # B×1×8×8
side_train = torch.randint(0, 2, (3000,)); X_train = blobs(3000, side_train)
side_test = torch.randint(0, 2, (500,)); X_test = blobs(500, side_test)
ae, model = AE(), LatentEps()
T = 100
betas = torch.linspace(1e-4, 0.05, T); alphas = 1 - betas; abar = torch.cumprod(alphas, 0)
# 4) Stage 1: train the autoencoder (reconstruction)
opt_ae = torch.optim.Adam(ae.parameters(), lr=2e-3)
for epoch in range(30):
for i in range(0, len(X_train), 64):
xb = X_train[i:i + 64]
l = F.mse_loss(ae(xb), xb); opt_ae.zero_grad(); l.backward(); opt_ae.step()
with torch.no_grad():
print(f"autoencoder test MSE = {F.mse_loss(ae(X_test), X_test).item():.4f}")
Z = ae.enc(X_train); mu, sd = Z.mean(0), Z.std(0); Z = (Z - mu) / sd # normalise the latent
# 5) Stage 2: DDPM diffusion in the latent space with the condition dropped at random (10%) to enable CFG
opt = torch.optim.Adam(model.parameters(), lr=2e-3)
for step in range(4000):
idx = torch.randint(0, len(Z), (256,))
z0, c = Z[idx], side_train[idx].clone()
c[torch.rand(256) < 0.1] = 2
t = torch.randint(0, T, (256,)); eps = torch.randn_like(z0)
zt = abar[t, None].sqrt() * z0 + (1 - abar[t, None]).sqrt() * eps
loss = F.mse_loss(model(zt, t, c), eps) # Forward Pass
opt.zero_grad(); loss.backward(); opt.step()
if step % 1000 == 0 or step == 3999:
print(f"step {step:4d} diffusion loss={loss.item():.4f}")
# 6) Evaluation: does the blob appear in the requested half? (classifier-free guidance, w = 3)
@torch.no_grad()
def generate(c, w=3.0):
n = len(c); z = torch.randn(n, 4)
for t in reversed(range(T)):
tt = torch.full((n,), t)
e = (1 + w) * model(z, tt, c) - w * model(z, tt, torch.full((n,), 2))
z = (z - betas[t] / (1 - abar[t]).sqrt() * e) / alphas[t].sqrt() + (betas[t].sqrt() * torch.randn_like(z) if t > 0 else 0)
return ae.dec(z * sd + mu).view(n, 1, 8, 8) # decode to pixels
c = torch.randint(0, 2, (1000,))
img = generate(c)
col = (img[:, 0].sum(1) * torch.arange(8).float()).sum(1) / img[:, 0].sum((1, 2)) # horizontal centre of mass
print(f"conditional accuracy (blob on requested side) = {((col > 3.5).long() == c).float().mean().item():.3f}")
# 7) Inference: ask for a blob in the right half
im = generate(torch.tensor([1]))[0, 0]
print("requested right half → peak at (row, col):", divmod(im.argmax().item(), 8))
This code was run on 2026-11-20 in 10.7 seconds (PyTorch (CPU), Python 3 – local build machine).
Sample output:
autoencoder test MSE = 0.0004 conditional accuracy (blob on requested side) = 1.000
Trainable parameters in the example: ae = 17,732, model = 21,812.
These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.
Try it and go deeper
Related architectures
Diffusion Models
Diffusion Models (DDPM)
They gradually corrupt the data by adding Gaussian noise over T steps, then a network learns to reverse this process by predicting the added noise, so a new example is g…
Variational Autoencoder
Variational Autoencoder (VAE)
A probabilistic generative model with a latent variable: an encoder that estimates an approximate distribution q(z x) and a decoder that represents p(x z), trained toget…
Vector-Quantised Variational Autoencoder
Vector Quantized Variational Autoencoder (VQ-VAE)
An autoencoder that quantises the encoder's output to the nearest vector in a learned codebook, so the representation becomes discrete codes; an autoregressive prior is…
Cross-attention
Cross-Attention (Cross-Attn)
Attention in which the queries come from one sequence while the keys and values come from another sequence or modality, so each element of the first reads what it needs…
U-Net
U-Net
A U-shaped segmentation architecture: a contracting path that captures context and an expanding path that restores spatial resolution, with skip connections that carry f…
Flow Matching
Flow Matching (FM)
It learns the velocity field of an ordinary differential equation that carries the noise distribution to the data distribution, by regressing directly on the velocities…
No results match the filters.
Research ideas
- Fine-tune an LDM on medical images with differential privacy and measure the benefit for segmentation models.
Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.
Test your understanding
What is the core idea of LDM?
Why? It reduces computation sharply.
How does text enter the U-Net in LDM?
Why? The text is encoded by $\tau_\theta$.
With $w=0$ in CFG, what does the prediction become?
Why? $(1+0)\epsilon(c)-0$.
References
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 10674–10685. https://doi.org/10.1109/CVPR52688.2022.01042
Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS) 33. https://arxiv.org/abs/2006.11239
Peebles, W., & Xie, S. (2023). Scalable diffusion models with transformers. IEEE/CVF International Conference on Computer Vision (ICCV), 4172–4182. https://doi.org/10.1109/ICCV51070.2023.00387
Song, J., Meng, C., & Ermon, S. (2021). Denoising diffusion implicit models. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2010.02502
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., et al. (2021). Learning transferable visual models from natural language supervision. International Conference on Machine Learning (ICML), PMLR 139, 8748–8763. https://arxiv.org/abs/2103.00020
