IntermediateHybrid ArchitecturesTransformer Architectures

U-Net with Transformers (U-Net+Transformer)

U-Net + Transformer

A family of segmentation models that keeps the U-Net shape (an encoder–decoder with skip connections) and introduces the transformer somewhere: the bottleneck, the whole encoder, or the whole architecture (a pure transformer model in a U shape).

Overview

NameU-Net with Transformers – U-Net + Transformer
Year introduced2021
ResearchersA family of designs; examples: Chen and colleagues (TransUNet 2021), Hatamizadeh and colleagues (UNETR and Swin UNETR), Cao and colleagues (Swin-Unet)
Original paperSwin-Unet: Unet-like pure transformer for medical image segmentation DOI arXiv
Data typeImages
DomainsComputer vision, Medical image analysis
Computational costU-Net + attention over the bottleneck tokens.
Data sizeMedium; pretraining helps.

Problem it solves

Convolutional U-Net is strong on detail but its receptive field is limited, so it errs when the decision needs wide anatomical context; a transformer alone loses fine spatial detail.

Background and evolution

What came before: U-Net, Transformer, Vision Transformer

  1. 2015

    U-Net. [ref]

  2. 2021

    TransUNet: ViT in the encoder after a CNN. [ref]

  3. 2022

    UNETR: a pure ViT as the encoder for 3D images. [ref]

  4. 2022

    Swin UNETR: a hierarchical Swin encoder. [ref]

  5. 2023

    Swin-Unet: an encoder and decoder made of Swin blocks (ECCV 2022 workshop). [ref]

Core idea

The main options: (1) a transformer in the bottleneck only (the cheapest, and the one shown in the code), (2) a hybrid CNN→ViT encoder (TransUNet), (3) a pure ViT encoder (UNETR), (4) a hierarchical windowed encoder (Swin UNETR), (5) a transformer in the encoder and the decoder (Swin-Unet). Skip connections remain to restore detail.

Example: Pancreas segmentation: its shape varies a lot and needs the context of neighbouring organs, so global attention in the bottleneck helps.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
CNN encoder blocks
Transformer (bottleneck or encoder)
Skip connections
CNN decoder
Segmentation head

Equations

1. Dice coefficient and soft Dice loss

$$\mathrm{Dice}=\frac{2|P\cap G|}{|P|+|G|},\qquad \mathcal{L}_{Dice}=1-\frac{2\sum_i p_ig_i+\epsilon}{\sum_ip_i+\sum_ig_i+\epsilon}$$

Symbols
$P,G$Prediction and ground truth
$p_i$Foreground probability for pixel i
Dimensions
A value between 0 and 1
Numerical example
$|P|=80$, $|G|=70$, $|P\cap G|=65$ ⇒ Dice $=130/150\approx0.867$.
Relation to the architecture
A common metric and loss in medical segmentation because they handle the imbalance between foreground and background well.

2. Attention cost by transformer position

$$\text{cost}\propto N^2d,\qquad N=\frac{HW}{4^{\ell}}$$

Symbols
$\ell$Number of reduction levels before the transformer
Dimensions
Approximate
Numerical example
A $256\times256$ image: at $\ell=1$ (128×128): $N=16384$, and at $\ell=4$ (16×16): $N=256$, i.e. $4096$ times cheaper in the attention term.
Relation to the architecture
It explains why the transformer is usually placed in the deep layers.

How it works, step by step

  1. Encode with convolutions and reduction.
  2. Apply the transformer to the deepest representation.
  3. Decode with skip connections.
  4. Train with CE + Dice.

Strengths and limitations

Strengths

  • Global context with local detail.
  • Improvements on tasks that need long-range relationships.

Limitations

  • More parameters and memory.
  • Gains on small data are not guaranteed; a well-tuned U-Net (nnU-Net) is very strong.

When to use it, and when not?

Good for

  • Large organs of variable shape.
  • When U-Net fails because of context.

Not suited for

  • Very little data without pretraining.

Applications

DomainUse
Medical image analysisSegmenting organs and tumours in CT and MRI.
Computer visionSemantic segmentation of natural images.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

Python (PyTorch)
# U-Net with a transformer in the bottleneck — segmentation
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
def block(ci, co):
    return nn.Sequential(nn.Conv2d(ci, co, 3, padding=1), nn.BatchNorm2d(co), nn.ReLU(),
                         nn.Conv2d(co, co, 3, padding=1), nn.BatchNorm2d(co), nn.ReLU())

class UNetTransformerBottleneck(nn.Module):
    """A classic U-Net with a transformer in the bottleneck: convolutions for local detail and attention for global context."""
    def __init__(self, k=2, d=32):
        super().__init__()
        self.e1, self.e2, self.e3 = block(1, 8), block(8, 16), block(16, d)            # 32 → 16 → 8
        self.pos = nn.Parameter(torch.randn(1, 64, d) * 0.02)                        # 8×8 = 64 tokens
        layer = nn.TransformerEncoderLayer(d, 4, 2 * d, dropout=0.0, batch_first=True, norm_first=True)
        self.tf = nn.TransformerEncoder(layer, 2, enable_nested_tensor=False)
        self.up2, self.d2 = nn.ConvTranspose2d(d, 16, 2, 2), block(32, 16)
        self.up1, self.d1 = nn.ConvTranspose2d(16, 8, 2, 2), block(16, 8)
        self.out = nn.Conv2d(8, k, 1)
    def forward(self, x):
        e1 = self.e1(x); e2 = self.e2(F.max_pool2d(e1, 2)); e3 = self.e3(F.max_pool2d(e2, 2))
        B, C, H, W = e3.shape
        t = self.tf(e3.flatten(2).transpose(1, 2) + self.pos)                        # global attention over the bottleneck
        m = t.transpose(1, 2).reshape(B, C, H, W) + e3
        d2 = self.d2(torch.cat([self.up2(m), e2], 1))                                 # skip connections
        d1 = self.d1(torch.cat([self.up1(d2), e1], 1))
        return self.out(d1)

# 3) Prepare the data
# 32×32 images with one circle at a random position and radius and strong noise; the target is the circle's mask
def make(n, S=32):
    yy, xx = torch.meshgrid(torch.arange(S), torch.arange(S), indexing='ij')
    cx, cy, r = torch.randint(8, 24, (n,)), torch.randint(8, 24, (n,)), torch.randint(4, 8, (n,))
    mask = ((xx[None] - cx[:, None, None]) ** 2 + (yy[None] - cy[:, None, None]) ** 2 < r[:, None, None] ** 2).long()
    X = mask[:, None].float() + 0.4 * torch.randn(n, 1, S, S)
    return X, mask
X_train, y_train = make(384)
X_test, y_test = make(96)

def dice(pred, target):
    inter = ((pred == 1) & (target == 1)).sum().item()
    return 2 * inter / max(1, (pred == 1).sum().item() + (target == 1).sum().item())
model = UNetTransformerBottleneck()

# 4) Loss function (cross-entropy + soft Dice, common in medical segmentation) and optimizer
def soft_dice_loss(logits, target):
    p = logits.softmax(1)[:, 1]; t = target.float()
    return 1 - (2 * (p * t).sum() + 1) / (p.sum() + t.sum() + 1)
opt = torch.optim.AdamW(model.parameters(), lr=2e-3, weight_decay=1e-4)

# 5) Training loop
EPOCHS, BATCH = 15, 16
for epoch in range(EPOCHS):
    model.train()
    perm, tot = torch.randperm(len(X_train)), 0.0
    for i in range(0, len(X_train), BATCH):
        idx = perm[i:i + BATCH]
        logits = model(X_train[idx])                                       # Forward Pass: B×2×H×W
        loss = F.cross_entropy(logits, y_train[idx]) + soft_dice_loss(logits, y_train[idx])
        opt.zero_grad(); loss.backward(); opt.step()
        tot += loss.item() * len(idx)
    if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
        print(f"epoch {epoch:3d}  loss={tot / len(X_train):.4f}")

# 6) Evaluation with the Dice coefficient on the test images
model.eval()
with torch.no_grad():
    pred = model(X_test).argmax(1)
print(f"test Dice = {dice(pred, y_test):.3f}")

# 7) Inference: the mask of a new image
with torch.no_grad():
    m = model(X_test[:1]).argmax(1)[0]
print("predicted foreground pixels:", int(m.sum()), " true:", int(y_test[0].sum()))

This code was run on 2026-11-20 in 11.0 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

test Dice = 0.988

Trainable parameters in the example: model = 48,786.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Basic2015

U-Net

U-Net

A U-shaped segmentation architecture: a contracting path that captures context and an expanding path that restores spatial resolution, with skip connections that carry f…

FoundationsHybrid Architectures

Intermediate2021

TransUNet

TransUNet

A medical segmentation model that uses a hybrid encoder: a CNN (ResNet-50) extracts a feature map, which is split into patches and passed through a ViT, followed by a ca…

Hybrid ArchitecturesTransformer Architectures

Advanced2022

UNETR

UNETR

A 3D segmentation model whose encoder is a pure ViT on 16³ volumetric patches; representations are taken from its layers (3, 6, 9, 12), reshaped and upsampled with trans…

Hybrid ArchitecturesTransformer Architectures

Advanced2022

Swin UNETR

Swin UNETR (Swin-UNETR)

A volumetric segmentation model whose encoder is a hierarchical Swin Transformer (shifted local windows and patch merging) that gives features at five scales, connected…

Hybrid ArchitecturesTransformer Architectures

Basic2018

Attention U-Net

Attention U-Net

A U-Net with attention gates on the skip connections that learn to highlight the relevant regions and suppress the background before the features are merged.

Hybrid Architectures

Research ideas

  • Compare transformer positions in U-Net for liver tumour segmentation, with nnU-Net as a baseline.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    Why is the transformer usually placed in the bottleneck?

  2. Mathematical

    $|P|=50$, $|G|=50$, $|P\cap G|=40$: what is the Dice?

  3. Concept

    What role do skip connections play in this family?

References

  1. Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., & Wang, M. (2023). Swin-Unet: Unet-like pure transformer for medical image segmentation. Computer Vision – ECCV 2022 Workshops, LNCS 13803, 205–218. https://doi.org/10.1007/978-3-031-25066-8_9

    DOIarXivAutomatically verified via Crossref – 2026-11-20

  2. Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021). TransUNet: Transformers make strong encoders for medical image segmentation. arXiv preprint. https://arxiv.org/abs/2102.04306

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  3. Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H. R., & Xu, D. (2022). UNETR: Transformers for 3D medical image segmentation. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 1748–1758. https://doi.org/10.1109/WACV51458.2022.00181

    DOIarXivAutomatically verified via Crossref – 2026-11-20

  4. Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H. R., & Xu, D. (2022). Swin UNETR: Swin transformers for semantic segmentation of brain tumors in MRI images. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries (BrainLes 2021), LNCS 12962, 272–284. https://doi.org/10.1007/978-3-031-08999-2_22

    DOIarXivAutomatically verified via Crossref – 2026-11-20

  5. Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention (MICCAI), LNCS 9351, 234–241. https://doi.org/10.1007/978-3-319-24574-4_28

    DOIarXivAutomatically verified via Crossref – 2026-11-20

  6. Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18(2), 203–211. https://doi.org/10.1038/s41592-020-01008-z

    DOIAutomatically verified via Crossref – 2026-11-20