U-Net with Transformers (U-Net+Transformer)
U-Net + Transformer
A family of segmentation models that keeps the U-Net shape (an encoder–decoder with skip connections) and introduces the transformer somewhere: the bottleneck, the whole encoder, or the whole architecture (a pure transformer model in a U shape).
Overview
| Name | U-Net with Transformers – U-Net + Transformer |
|---|---|
| Year introduced | 2021 |
| Researchers | A family of designs; examples: Chen and colleagues (TransUNet 2021), Hatamizadeh and colleagues (UNETR and Swin UNETR), Cao and colleagues (Swin-Unet) |
| Original paper | Swin-Unet: Unet-like pure transformer for medical image segmentation DOI arXiv |
| Data type | Images |
| Domains | Computer vision, Medical image analysis |
| Computational cost | U-Net + attention over the bottleneck tokens. |
| Data size | Medium; pretraining helps. |
Problem it solves
Convolutional U-Net is strong on detail but its receptive field is limited, so it errs when the decision needs wide anatomical context; a transformer alone loses fine spatial detail.
Background and evolution
What came before: U-Net, Transformer, Vision Transformer
Core idea
The main options: (1) a transformer in the bottleneck only (the cheapest, and the one shown in the code), (2) a hybrid CNN→ViT encoder (TransUNet), (3) a pure ViT encoder (UNETR), (4) a hierarchical windowed encoder (Swin UNETR), (5) a transformer in the encoder and the decoder (Swin-Unet). Skip connections remain to restore detail.
Interactive diagram
A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.
Select a layer in the diagram.
Components
| Component | Role |
|---|---|
| CNN encoder blocks | |
| Transformer (bottleneck or encoder) | |
| Skip connections | |
| CNN decoder | |
| Segmentation head |
Equations
1. Dice coefficient and soft Dice loss
| $P,G$ | Prediction and ground truth |
| $p_i$ | Foreground probability for pixel i |
2. Attention cost by transformer position
| $\ell$ | Number of reduction levels before the transformer |
How it works, step by step
- Encode with convolutions and reduction.
- Apply the transformer to the deepest representation.
- Decode with skip connections.
- Train with CE + Dice.
Strengths and limitations
Strengths
- Global context with local detail.
- Improvements on tasks that need long-range relationships.
Limitations
- More parameters and memory.
- Gains on small data are not guaranteed; a well-tuned U-Net (nnU-Net) is very strong.
When to use it, and when not?
Good for
- Large organs of variable shape.
- When U-Net fails because of context.
Not suited for
- Very little data without pretraining.
Applications
| Domain | Use |
|---|---|
| Medical image analysis | Segmenting organs and tumours in CT and MRI. |
| Computer vision | Semantic segmentation of natural images. |
PyTorch code
A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.
# U-Net with a transformer in the bottleneck — segmentation
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
# 2) Define the model
def block(ci, co):
return nn.Sequential(nn.Conv2d(ci, co, 3, padding=1), nn.BatchNorm2d(co), nn.ReLU(),
nn.Conv2d(co, co, 3, padding=1), nn.BatchNorm2d(co), nn.ReLU())
class UNetTransformerBottleneck(nn.Module):
"""A classic U-Net with a transformer in the bottleneck: convolutions for local detail and attention for global context."""
def __init__(self, k=2, d=32):
super().__init__()
self.e1, self.e2, self.e3 = block(1, 8), block(8, 16), block(16, d) # 32 → 16 → 8
self.pos = nn.Parameter(torch.randn(1, 64, d) * 0.02) # 8×8 = 64 tokens
layer = nn.TransformerEncoderLayer(d, 4, 2 * d, dropout=0.0, batch_first=True, norm_first=True)
self.tf = nn.TransformerEncoder(layer, 2, enable_nested_tensor=False)
self.up2, self.d2 = nn.ConvTranspose2d(d, 16, 2, 2), block(32, 16)
self.up1, self.d1 = nn.ConvTranspose2d(16, 8, 2, 2), block(16, 8)
self.out = nn.Conv2d(8, k, 1)
def forward(self, x):
e1 = self.e1(x); e2 = self.e2(F.max_pool2d(e1, 2)); e3 = self.e3(F.max_pool2d(e2, 2))
B, C, H, W = e3.shape
t = self.tf(e3.flatten(2).transpose(1, 2) + self.pos) # global attention over the bottleneck
m = t.transpose(1, 2).reshape(B, C, H, W) + e3
d2 = self.d2(torch.cat([self.up2(m), e2], 1)) # skip connections
d1 = self.d1(torch.cat([self.up1(d2), e1], 1))
return self.out(d1)
# 3) Prepare the data
# 32×32 images with one circle at a random position and radius and strong noise; the target is the circle's mask
def make(n, S=32):
yy, xx = torch.meshgrid(torch.arange(S), torch.arange(S), indexing='ij')
cx, cy, r = torch.randint(8, 24, (n,)), torch.randint(8, 24, (n,)), torch.randint(4, 8, (n,))
mask = ((xx[None] - cx[:, None, None]) ** 2 + (yy[None] - cy[:, None, None]) ** 2 < r[:, None, None] ** 2).long()
X = mask[:, None].float() + 0.4 * torch.randn(n, 1, S, S)
return X, mask
X_train, y_train = make(384)
X_test, y_test = make(96)
def dice(pred, target):
inter = ((pred == 1) & (target == 1)).sum().item()
return 2 * inter / max(1, (pred == 1).sum().item() + (target == 1).sum().item())
model = UNetTransformerBottleneck()
# 4) Loss function (cross-entropy + soft Dice, common in medical segmentation) and optimizer
def soft_dice_loss(logits, target):
p = logits.softmax(1)[:, 1]; t = target.float()
return 1 - (2 * (p * t).sum() + 1) / (p.sum() + t.sum() + 1)
opt = torch.optim.AdamW(model.parameters(), lr=2e-3, weight_decay=1e-4)
# 5) Training loop
EPOCHS, BATCH = 15, 16
for epoch in range(EPOCHS):
model.train()
perm, tot = torch.randperm(len(X_train)), 0.0
for i in range(0, len(X_train), BATCH):
idx = perm[i:i + BATCH]
logits = model(X_train[idx]) # Forward Pass: B×2×H×W
loss = F.cross_entropy(logits, y_train[idx]) + soft_dice_loss(logits, y_train[idx])
opt.zero_grad(); loss.backward(); opt.step()
tot += loss.item() * len(idx)
if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
print(f"epoch {epoch:3d} loss={tot / len(X_train):.4f}")
# 6) Evaluation with the Dice coefficient on the test images
model.eval()
with torch.no_grad():
pred = model(X_test).argmax(1)
print(f"test Dice = {dice(pred, y_test):.3f}")
# 7) Inference: the mask of a new image
with torch.no_grad():
m = model(X_test[:1]).argmax(1)[0]
print("predicted foreground pixels:", int(m.sum()), " true:", int(y_test[0].sum()))
This code was run on 2026-11-20 in 11.0 seconds (PyTorch (CPU), Python 3 – local build machine).
Sample output:
test Dice = 0.988
Trainable parameters in the example: model = 48,786.
These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.
Try it and go deeper
- Transformer Visualizer
- Medical Image Analysis Lab Another section of the site
Related architectures
U-Net
U-Net
A U-shaped segmentation architecture: a contracting path that captures context and an expanding path that restores spatial resolution, with skip connections that carry f…
TransUNet
TransUNet
A medical segmentation model that uses a hybrid encoder: a CNN (ResNet-50) extracts a feature map, which is split into patches and passed through a ViT, followed by a ca…
UNETR
UNETR
A 3D segmentation model whose encoder is a pure ViT on 16³ volumetric patches; representations are taken from its layers (3, 6, 9, 12), reshaped and upsampled with trans…
Swin UNETR
Swin UNETR (Swin-UNETR)
A volumetric segmentation model whose encoder is a hierarchical Swin Transformer (shifted local windows and patch merging) that gives features at five scales, connected…
Attention U-Net
Attention U-Net
A U-Net with attention gates on the skip connections that learn to highlight the relevant regions and suppress the background before the features are merged.
No results match the filters.
Research ideas
- Compare transformer positions in U-Net for liver tumour segmentation, with nnU-Net as a baseline.
Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.
Test your understanding
Why is the transformer usually placed in the bottleneck?
Why? The cost is quadratic in the number of tokens.
$|P|=50$, $|G|=50$, $|P\cap G|=40$: what is the Dice?
Why? $80/100$.
What role do skip connections play in this family?
Why? As in the original U-Net.
References
Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., & Wang, M. (2023). Swin-Unet: Unet-like pure transformer for medical image segmentation. Computer Vision – ECCV 2022 Workshops, LNCS 13803, 205–218. https://doi.org/10.1007/978-3-031-25066-8_9
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021). TransUNet: Transformers make strong encoders for medical image segmentation. arXiv preprint. https://arxiv.org/abs/2102.04306
Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H. R., & Xu, D. (2022). UNETR: Transformers for 3D medical image segmentation. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 1748–1758. https://doi.org/10.1109/WACV51458.2022.00181
Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H. R., & Xu, D. (2022). Swin UNETR: Swin transformers for semantic segmentation of brain tumors in MRI images. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries (BrainLes 2021), LNCS 12962, 272–284. https://doi.org/10.1007/978-3-031-08999-2_22
Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention (MICCAI), LNCS 9351, 234–241. https://doi.org/10.1007/978-3-319-24574-4_28
Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18(2), 203–211. https://doi.org/10.1038/s41592-020-01008-z
