BasicFoundationsHybrid Architectures

U-Net

A U-shaped segmentation architecture: a contracting path that captures context and an expanding path that restores spatial resolution, with skip connections that carry fine detail between matching levels.

Overview

NameU-Net
Year introduced2015
ResearchersOlaf Ronneberger, Philipp Fischer, Thomas Brox
InstitutionUniversity of Freiburg
Original paperU-Net: Convolutional networks for biomedical image segmentation DOI arXiv
Data typeImages
DomainsComputer vision, Medical image analysis

Problem it solves

Medical image segmentation requires precise pixel-level delineation with few labelled images.

Background and evolution

What came before: Convolutional Neural Network, Fully Convolutional Networks

  1. 2015

    It was introduced at MICCAI 2015 for cell segmentation in microscopy images. [ref]

  2. 2018

    Attention U-Net added attention gates on the skip connections. [ref]

  3. 2021

    nnU-Net automated tuning U-Net's settings for each dataset. [ref]

Core idea

The encoder reduces resolution and doubles the channels, capturing "what" is in the image. The decoder raises the resolution gradually to determine "where". Skip connections merge the encoder's high-resolution maps with the decoder's maps, so fine edges are not lost.

Example: On a 128×128 brain MRI image, U-Net produces a mask of the same size that marks for each pixel whether it is tumour or healthy tissue.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
InputA $C\times H\times W$ image
Encoder(Conv-ReLU)×2 blocks, then MaxPool
BottleneckThe deepest level with the largest number of channels
DecoderUpsampling (ConvTranspose or Upsample), then a convolution
Skip ConnectionsConcatenating encoder maps with the decoder
OutputA 1×1 convolution with as many outputs as classes
Loss FunctionCross-entropy, Dice, or a mix of both

Equations

1. Dice coefficient

$$\mathrm{Dice}(P,G)=\frac{2|P\cap G|}{|P|+|G|}$$

Symbols
$P$Predicted mask
$G$True mask
Dimensions
Between 0 and 1
Numerical example
$|P|=50,\;|G|=40,\;|P\cap G|=30\Rightarrow \mathrm{Dice}=60/90\approx0.667$.
Relation to the architecture
The most widely used metric for evaluating medical segmentation, and the source of the Dice loss.

2. Differentiable Dice loss

$$\mathcal{L}_{Dice}=1-\frac{2\sum_i p_i g_i+\epsilon}{\sum_i p_i+\sum_i g_i+\epsilon}$$

Symbols
$p_i$Probability of pixel i
$g_i$Label 0 or 1
$\epsilon$A small constant for stability
Dimensions
A scalar
Numerical example
If the two masks match exactly, $\mathcal{L}_{Dice}=0$.
Relation to the architecture
Useful when the target region is small compared with the background.

How it works, step by step

  1. The image enters the encoder.
  2. Each level doubles the channels and reduces the resolution.
  3. The representation reaches the bottleneck.
  4. The decoder raises the resolution level by level.
  5. At each level the matching encoder maps are concatenated.
  6. A 1×1 convolution gives the class scores for each pixel.

Strengths and limitations

Strengths

  • High segmentation accuracy with little data.
  • A flexible structure that extends easily to 3D.
  • A very strong baseline in medical imaging.

Limitations

  • The receptive field is limited by convolution; very long-range relationships need additions.
  • High memory use in 3D versions.

When to use it, and when not?

Good for

  • Segmentation of organs, tumours and cells.
  • Any task whose output is an image of the same size (such as denoising).

Not suited for

  • Image-level classification only, where a simpler CNN is enough.

Applications

DomainUse
Medical image analysisSegmenting tumours and organs in CT and MRI.
Computer visionSemantic segmentation and satellite images.
Generative AIThe base network for denoising in diffusion models.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

Python (PyTorch)
# A miniature U-Net for binary segmentation
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
def block(ci, co):
    return nn.Sequential(nn.Conv2d(ci, co, 3, padding=1), nn.BatchNorm2d(co), nn.ReLU(),
                         nn.Conv2d(co, co, 3, padding=1), nn.BatchNorm2d(co), nn.ReLU())
class UNet(nn.Module):
    def __init__(self, c=1, k=2):
        super().__init__()
        self.e1, self.e2, self.mid = block(c, 8), block(8, 16), block(16, 32)
        self.up2, self.d2 = nn.ConvTranspose2d(32, 16, 2, 2), block(32, 16)
        self.up1, self.d1 = nn.ConvTranspose2d(16, 8, 2, 2), block(16, 8)
        self.out = nn.Conv2d(8, k, 1)
    def forward(self, x):
        e1 = self.e1(x); e2 = self.e2(F.max_pool2d(e1, 2)); m = self.mid(F.max_pool2d(e2, 2))
        d2 = self.d2(torch.cat([self.up2(m), e2], 1))      # skip connection
        d1 = self.d1(torch.cat([self.up1(d2), e1], 1))
        return self.out(d1)                                # B×K×H×W

# 3) Prepare the data
# 32×32 images with one circle at a random position and radius; the target is the circle's mask
def make(n, S=32):
    yy, xx = torch.meshgrid(torch.arange(S), torch.arange(S), indexing='ij')
    cx, cy, r = torch.randint(8, 24, (n,)), torch.randint(8, 24, (n,)), torch.randint(4, 8, (n,))
    mask = ((xx[None] - cx[:, None, None]) ** 2 + (yy[None] - cy[:, None, None]) ** 2 < r[:, None, None] ** 2).long()
    X = mask[:, None].float() + 0.4 * torch.randn(n, 1, S, S)
    return X, mask
X_train, y_train = make(256)
X_test, y_test = make(64)
model = UNet()

def dice(pred, target):
    inter = ((pred == 1) & (target == 1)).sum().item()
    return 2 * inter / max(1, (pred == 1).sum().item() + (target == 1).sum().item())

# 4) Loss function and optimizer (per-pixel cross-entropy)
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=1e-2)

# 5) Training loop
for epoch in range(6):
    model.train()
    for i in range(0, len(X_train), 32):
        xb, yb = X_train[i:i + 32], y_train[i:i + 32]
        loss = loss_fn(model(xb), yb)        # forward pass: B×2×H×W against B×H×W
        opt.zero_grad(); loss.backward(); opt.step()
    print(f"epoch {epoch}  loss={loss.item():.4f}")

# 6) Evaluation with the Dice coefficient
model.eval()
with torch.no_grad():
    pred = model(X_test).argmax(1)
print(f"test Dice = {dice(pred, y_test):.3f}")

# 7) Inference: the mask of one image
with torch.no_grad():
    m = model(X_test[:1]).argmax(1)[0]
print("predicted foreground pixels:", int(m.sum()))

This code was run on 2026-11-20 in 4.0 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

test Dice = 0.972

Trainable parameters in the example: model = 29,650.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Basic2018

Attention U-Net

Attention U-Net

A U-Net with attention gates on the skip connections that learn to highlight the relevant regions and suppress the background before the features are merged.

Hybrid Architectures

Intermediate2021

TransUNet

TransUNet

A medical segmentation model that uses a hybrid encoder: a CNN (ResNet-50) extracts a feature map, which is split into patches and passed through a ViT, followed by a ca…

Hybrid ArchitecturesTransformer Architectures

Advanced2022

UNETR

UNETR

A 3D segmentation model whose encoder is a pure ViT on 16³ volumetric patches; representations are taken from its layers (3, 6, 9, 12), reshaped and upsampled with trans…

Hybrid ArchitecturesTransformer Architectures

Advanced2022

Swin UNETR

Swin UNETR (Swin-UNETR)

A volumetric segmentation model whose encoder is a hierarchical Swin Transformer (shifted local windows and patch merging) that gives features at five scales, connected…

Hybrid ArchitecturesTransformer Architectures

Advanced2020

Diffusion Models

Diffusion Models (DDPM)

They gradually corrupt the data by adding Gaussian noise over T steps, then a network learns to reverse this process by predicting the added noise, so a new example is g…

Generative Networks

Research ideas

  • Compare U-Net, nnU-Net and Swin-UNETR on a single prostate dataset with a patient-level split protocol.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    What is the function of the skip connections in U-Net?

  2. Mathematical

    What is the Dice value if $|P|=|G|=20$ and $|P\cap G|=10$?

  3. Concept

    In which field did U-Net first appear?

References

  1. Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention (MICCAI), LNCS 9351, 234–241. https://doi.org/10.1007/978-3-319-24574-4_28

    DOIarXivAutomatically verified via Crossref – 2026-11-20

  2. Oktay, O., Schlemper, J., Le Folgoc, L., Lee, M., Heinrich, M., Misawa, K., et al. (2018). Attention U-Net: Learning where to look for the pancreas. Medical Imaging with Deep Learning (MIDL). https://arxiv.org/abs/1804.03999

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  3. Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18(2), 203–211. https://doi.org/10.1038/s41592-020-01008-z

    DOIAutomatically verified via Crossref – 2026-11-20