U-Net
A U-shaped segmentation architecture: a contracting path that captures context and an expanding path that restores spatial resolution, with skip connections that carry fine detail between matching levels.
Overview
| Name | U-Net |
|---|---|
| Year introduced | 2015 |
| Researchers | Olaf Ronneberger, Philipp Fischer, Thomas Brox |
| Institution | University of Freiburg |
| Original paper | U-Net: Convolutional networks for biomedical image segmentation DOI arXiv |
| Data type | Images |
| Domains | Computer vision, Medical image analysis |
Problem it solves
Medical image segmentation requires precise pixel-level delineation with few labelled images.
Background and evolution
What came before: Convolutional Neural Network, Fully Convolutional Networks
Core idea
The encoder reduces resolution and doubles the channels, capturing "what" is in the image. The decoder raises the resolution gradually to determine "where". Skip connections merge the encoder's high-resolution maps with the decoder's maps, so fine edges are not lost.
Interactive diagram
A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.
Select a layer in the diagram.
Components
| Component | Role |
|---|---|
| Input | A $C\times H\times W$ image |
| Encoder | (Conv-ReLU)×2 blocks, then MaxPool |
| Bottleneck | The deepest level with the largest number of channels |
| Decoder | Upsampling (ConvTranspose or Upsample), then a convolution |
| Skip Connections | Concatenating encoder maps with the decoder |
| Output | A 1×1 convolution with as many outputs as classes |
| Loss Function | Cross-entropy, Dice, or a mix of both |
Equations
1. Dice coefficient
| $P$ | Predicted mask |
| $G$ | True mask |
2. Differentiable Dice loss
| $p_i$ | Probability of pixel i |
| $g_i$ | Label 0 or 1 |
| $\epsilon$ | A small constant for stability |
How it works, step by step
- The image enters the encoder.
- Each level doubles the channels and reduces the resolution.
- The representation reaches the bottleneck.
- The decoder raises the resolution level by level.
- At each level the matching encoder maps are concatenated.
- A 1×1 convolution gives the class scores for each pixel.
Strengths and limitations
Strengths
- High segmentation accuracy with little data.
- A flexible structure that extends easily to 3D.
- A very strong baseline in medical imaging.
Limitations
- The receptive field is limited by convolution; very long-range relationships need additions.
- High memory use in 3D versions.
When to use it, and when not?
Good for
- Segmentation of organs, tumours and cells.
- Any task whose output is an image of the same size (such as denoising).
Not suited for
- Image-level classification only, where a simpler CNN is enough.
Applications
| Domain | Use |
|---|---|
| Medical image analysis | Segmenting tumours and organs in CT and MRI. |
| Computer vision | Semantic segmentation and satellite images. |
| Generative AI | The base network for denoising in diffusion models. |
PyTorch code
A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.
# A miniature U-Net for binary segmentation
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
# 2) Define the model
def block(ci, co):
return nn.Sequential(nn.Conv2d(ci, co, 3, padding=1), nn.BatchNorm2d(co), nn.ReLU(),
nn.Conv2d(co, co, 3, padding=1), nn.BatchNorm2d(co), nn.ReLU())
class UNet(nn.Module):
def __init__(self, c=1, k=2):
super().__init__()
self.e1, self.e2, self.mid = block(c, 8), block(8, 16), block(16, 32)
self.up2, self.d2 = nn.ConvTranspose2d(32, 16, 2, 2), block(32, 16)
self.up1, self.d1 = nn.ConvTranspose2d(16, 8, 2, 2), block(16, 8)
self.out = nn.Conv2d(8, k, 1)
def forward(self, x):
e1 = self.e1(x); e2 = self.e2(F.max_pool2d(e1, 2)); m = self.mid(F.max_pool2d(e2, 2))
d2 = self.d2(torch.cat([self.up2(m), e2], 1)) # skip connection
d1 = self.d1(torch.cat([self.up1(d2), e1], 1))
return self.out(d1) # B×K×H×W
# 3) Prepare the data
# 32×32 images with one circle at a random position and radius; the target is the circle's mask
def make(n, S=32):
yy, xx = torch.meshgrid(torch.arange(S), torch.arange(S), indexing='ij')
cx, cy, r = torch.randint(8, 24, (n,)), torch.randint(8, 24, (n,)), torch.randint(4, 8, (n,))
mask = ((xx[None] - cx[:, None, None]) ** 2 + (yy[None] - cy[:, None, None]) ** 2 < r[:, None, None] ** 2).long()
X = mask[:, None].float() + 0.4 * torch.randn(n, 1, S, S)
return X, mask
X_train, y_train = make(256)
X_test, y_test = make(64)
model = UNet()
def dice(pred, target):
inter = ((pred == 1) & (target == 1)).sum().item()
return 2 * inter / max(1, (pred == 1).sum().item() + (target == 1).sum().item())
# 4) Loss function and optimizer (per-pixel cross-entropy)
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=1e-2)
# 5) Training loop
for epoch in range(6):
model.train()
for i in range(0, len(X_train), 32):
xb, yb = X_train[i:i + 32], y_train[i:i + 32]
loss = loss_fn(model(xb), yb) # forward pass: B×2×H×W against B×H×W
opt.zero_grad(); loss.backward(); opt.step()
print(f"epoch {epoch} loss={loss.item():.4f}")
# 6) Evaluation with the Dice coefficient
model.eval()
with torch.no_grad():
pred = model(X_test).argmax(1)
print(f"test Dice = {dice(pred, y_test):.3f}")
# 7) Inference: the mask of one image
with torch.no_grad():
m = model(X_test[:1]).argmax(1)[0]
print("predicted foreground pixels:", int(m.sum()))
This code was run on 2026-11-20 in 4.0 seconds (PyTorch (CPU), Python 3 – local build machine).
Sample output:
test Dice = 0.972
Trainable parameters in the example: model = 29,650.
These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.
Try it and go deeper
- AI Model Atlas Another section of the site
- Medical Image Analysis Lab Another section of the site
Related architectures
Attention U-Net
Attention U-Net
A U-Net with attention gates on the skip connections that learn to highlight the relevant regions and suppress the background before the features are merged.
TransUNet
TransUNet
A medical segmentation model that uses a hybrid encoder: a CNN (ResNet-50) extracts a feature map, which is split into patches and passed through a ViT, followed by a ca…
UNETR
UNETR
A 3D segmentation model whose encoder is a pure ViT on 16³ volumetric patches; representations are taken from its layers (3, 6, 9, 12), reshaped and upsampled with trans…
Swin UNETR
Swin UNETR (Swin-UNETR)
A volumetric segmentation model whose encoder is a hierarchical Swin Transformer (shifted local windows and patch merging) that gives features at five scales, connected…
Diffusion Models
Diffusion Models (DDPM)
They gradually corrupt the data by adding Gaussian noise over T steps, then a network learns to reverse this process by predicting the added noise, so a new example is g…
No results match the filters.
Research ideas
- Compare U-Net, nnU-Net and Swin-UNETR on a single prostate dataset with a patient-level split protocol.
Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.
Test your understanding
What is the function of the skip connections in U-Net?
Why? The encoder's high-resolution maps are concatenated with the decoder's maps to restore precise boundaries.
What is the Dice value if $|P|=|G|=20$ and $|P\cap G|=10$?
Why? $2\times10/(20+20)=0.5$.
In which field did U-Net first appear?
Why? It was introduced at MICCAI 2015 for cell segmentation in microscopy images.
References
Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention (MICCAI), LNCS 9351, 234–241. https://doi.org/10.1007/978-3-319-24574-4_28
Oktay, O., Schlemper, J., Le Folgoc, L., Lee, M., Heinrich, M., Misawa, K., et al. (2018). Attention U-Net: Learning where to look for the pancreas. Medical Imaging with Deep Learning (MIDL). https://arxiv.org/abs/1804.03999
Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18(2), 203–211. https://doi.org/10.1038/s41592-020-01008-z
