AdvancedHybrid ArchitecturesTransformer Architectures

Swin UNETR (Swin-UNETR)

A volumetric segmentation model whose encoder is a hierarchical Swin Transformer (shifted local windows and patch merging) that gives features at five scales, connected to a convolutional decoder through skip connections; it was developed for brain tumour segmentation in the BraTS 2021 challenge.

Overview

NameSwin UNETR
Year introduced2022
ResearchersAli Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R. Roth, Daguang Xu
InstitutionNVIDIA
Original paperSwin UNETR: Swin transformers for semantic segmentation of brain tumors in MRI images DOI arXiv
Data typeImages
DomainsMedical image analysis
Computational costHigh, but linear in the number of tokens.
Data sizeHundreds of volumes (BraTS: more than 1000 training cases).

Problem it solves

UNETR is single-scale and loses detail, and global attention on volumes is costly. A hierarchical windowed encoder combines efficiency with multiple scales.

Background and evolution

What came before: Swin Transformer, UNETR, U-Net

  1. 2021

    Swin Transformer for natural images. [ref]

  2. 2022

    Swin UNETR at the BrainLes workshop (MICCAI 2021) for brain tumour segmentation. [ref]

Core idea

The encoder applies 3D Swin blocks (shifted cubic windows) with patch merging between stages, producing representations at decreasing resolutions. Each representation passes through a residual convolutional block and is then concatenated in the decoder at the same scale, just like U-Net.

Example: Segmenting the three tumour regions (the necrotic core, the enhancing tumour and the oedema) from four MRI modalities.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
3D patch partition
Swin blocks (W-MSA / SW-MSA)
Patch merging
Residual conv blocks on skips
CNN decoder

Equations

1. Windowed attention cost (3D)

$$\Omega=4\,hwd\,C^2+2\,M^3\,hwd\,C$$

Symbols
$M$Window size
$hwd$Number of tokens
Dimensions
Approximate
Numerical example
$hwd=32^3=32768$, $M=7$, $C=48$: the second term is $\approx2\times343\times32768\times48\approx1.08\times10^9$ versus $2(32768)^2(48)\approx1.03\times10^{11}$ for global attention.
Relation to the architecture
A cost linear in the number of tokens.

2. Dice coefficient and soft Dice loss

$$\mathrm{Dice}=\frac{2|P\cap G|}{|P|+|G|},\qquad \mathcal{L}_{Dice}=1-\frac{2\sum_i p_ig_i+\epsilon}{\sum_ip_i+\sum_ig_i+\epsilon}$$

Symbols
$P,G$Prediction and ground truth
$p_i$Foreground probability for pixel i
Dimensions
A value between 0 and 1
Numerical example
$|P|=80$, $|G|=70$, $|P\cap G|=65$ ⇒ Dice $=130/150\approx0.867$.
Relation to the architecture
A common metric and loss in medical segmentation because they handle the imbalance between foreground and background well.

How it works, step by step

  1. Split the volume into small patches.
  2. Apply the Swin stages with merging.
  3. Pass each scale through a convolutional block.
  4. Decode with concatenation.
  5. Train with Dice + CE.

Strengths and limitations

Strengths

  • Multiple scales with wide context.
  • The efficiency of windows.
  • Strong results in BraTS 2021, according to the authors.

Limitations

  • Implementation complexity.
  • Large GPU memory for volumes.

When to use it, and when not?

Good for

  • Volumetric segmentation that needs both context and detail.

Not suited for

  • Small 2D data for which U-Net is enough.

Applications

DomainUse
Medical image analysisSegmenting organs and tumours in CT and MRI.
Computer visionSemantic segmentation of natural images.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

The original version is 3D with 7³ windows and four stages; here is a 2D version with three stages and a 4×4 window that reuses the Swin blocks from the Swin page.

Python (PyTorch)
# A miniature 2D Swin UNETR: a hierarchical Swin encoder + a convolutional decoder
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
def window_partition(x, M):                     # B×H×W×C → (B·nW)×M²×C
    B, H, W, C = x.shape
    x = x.view(B, H // M, M, W // M, M, C).permute(0, 1, 3, 2, 4, 5)
    return x.reshape(-1, M * M, C)

def window_reverse(w, M, H, W):                 # (B·nW)×M²×C → B×H×W×C
    B = w.shape[0] // ((H // M) * (W // M))
    x = w.view(B, H // M, W // M, M, M, -1).permute(0, 1, 3, 2, 4, 5)
    return x.reshape(B, H, W, -1)

class WindowAttention(nn.Module):
    """Attention inside an M×M window with a learned relative position bias (a (2M−1)² table per head)."""
    def __init__(self, C, M, heads):
        super().__init__()
        self.h, self.dk, self.M = heads, C // heads, M
        self.qkv, self.proj = nn.Linear(C, 3 * C), nn.Linear(C, C)
        self.bias_table = nn.Parameter(torch.zeros((2 * M - 1) ** 2, heads))
        coords = torch.stack(torch.meshgrid(torch.arange(M), torch.arange(M), indexing='ij')).flatten(1)   # 2×M²
        rel = (coords[:, :, None] - coords[:, None, :]).permute(1, 2, 0) + (M - 1)                         # M²×M²×2
        self.register_buffer('rel_index', rel[..., 0] * (2 * M - 1) + rel[..., 1])
    def forward(self, x, mask=None):            # x: (B·nW)×M²×C ; mask: nW×M²×M²
        Bw, N, C = x.shape
        q, k, v = self.qkv(x).view(Bw, N, 3, self.h, self.dk).permute(2, 0, 3, 1, 4)
        a = q @ k.transpose(-2, -1) / self.dk ** 0.5
        a = a + self.bias_table[self.rel_index].permute(2, 0, 1)[None]
        if mask is not None:
            nW = mask.shape[0]
            a = a.view(Bw // nW, nW, self.h, N, N) + mask[None, :, None]
            a = a.view(Bw, self.h, N, N)
        return self.proj((a.softmax(-1) @ v).transpose(1, 2).reshape(Bw, N, C))

class SwinBlock(nn.Module):
    def __init__(self, C, H, W, heads, M=4, shift=0):
        super().__init__()
        self.H, self.W, self.M, self.s = H, W, M, shift
        self.n1, self.attn, self.n2 = nn.LayerNorm(C), WindowAttention(C, M, heads), nn.LayerNorm(C)
        self.mlp = nn.Sequential(nn.Linear(C, 2 * C), nn.GELU(), nn.Linear(2 * C, C))
        mask = None
        if shift:                               # a mask that prevents attention between regions that were not adjacent before the cyclic shift
            img = torch.zeros(1, H, W, 1); cnt = 0
            for hs in (slice(0, -M), slice(-M, -shift), slice(-shift, None)):
                for ws in (slice(0, -M), slice(-M, -shift), slice(-shift, None)):
                    img[:, hs, ws, :] = cnt; cnt += 1
            win = window_partition(img, M).squeeze(-1)                      # nW×M²
            mask = (win[:, None, :] != win[:, :, None]).float() * -100.0
        self.register_buffer('mask', mask)
    def forward(self, x):                       # x: B×H×W×C
        y = self.n1(x)
        if self.s: y = torch.roll(y, (-self.s, -self.s), (1, 2))            # SW-MSA: cyclic shift
        y = window_reverse(self.attn(window_partition(y, self.M), self.mask), self.M, self.H, self.W)
        if self.s: y = torch.roll(y, (self.s, self.s), (1, 2))
        x = x + y
        return x + self.mlp(self.n2(x))

class PatchMerging(nn.Module):
    """Merge every 2×2 group of neighbouring patches: spatial resolution halves and channels double."""
    def __init__(self, C):
        super().__init__()
        self.norm, self.red = nn.LayerNorm(4 * C), nn.Linear(4 * C, 2 * C, bias=False)
    def forward(self, x):
        x = torch.cat([x[:, 0::2, 0::2], x[:, 1::2, 0::2], x[:, 0::2, 1::2], x[:, 1::2, 1::2]], -1)
        return self.red(self.norm(x))

def conv(ci, co):
    return nn.Sequential(nn.Conv2d(ci, co, 3, padding=1), nn.InstanceNorm2d(co, affine=True), nn.LeakyReLU(0.01))

class SwinUNETR2D(nn.Module):
    """A hierarchical Swin encoder (regular then shifted windows + patch merging) feeding a convolutional decoder with skip connections at every scale."""
    def __init__(self, k=2, C=16, M=4):
        super().__init__()
        self.embed = nn.Conv2d(1, C, 2, 2)                                               # 32 → 16×16×C
        self.s1 = nn.Sequential(SwinBlock(C, 16, 16, 2, M, 0), SwinBlock(C, 16, 16, 2, M, 2))
        self.m1 = PatchMerging(C)                                                        # → 8×8×2C
        self.s2 = nn.Sequential(SwinBlock(2 * C, 8, 8, 4, M, 0), SwinBlock(2 * C, 8, 8, 4, M, 2))
        self.m2 = PatchMerging(2 * C)                                                    # → 4×4×4C
        self.s3 = SwinBlock(4 * C, 4, 4, 4, M, 0)
        self.enc0 = conv(1, C)                                                           # scale of the original image
        self.up3, self.dec3 = nn.ConvTranspose2d(4 * C, 2 * C, 2, 2), conv(4 * C, 2 * C)
        self.up2, self.dec2 = nn.ConvTranspose2d(2 * C, C, 2, 2), conv(2 * C, C)
        self.up1, self.dec1 = nn.ConvTranspose2d(C, C, 2, 2), conv(2 * C, C)
        self.out = nn.Conv2d(C, k, 1)
    def forward(self, x):
        h1 = self.s1(self.embed(x).permute(0, 2, 3, 1))                                  # B×16×16×C
        h2 = self.s2(self.m1(h1)); h3 = self.s3(self.m2(h2))
        t = lambda h: h.permute(0, 3, 1, 2)
        d = self.dec3(torch.cat([self.up3(t(h3)), t(h2)], 1))
        d = self.dec2(torch.cat([self.up2(d), t(h1)], 1))
        d = self.dec1(torch.cat([self.up1(d), self.enc0(x)], 1))
        return self.out(d)

# 3) Prepare the data
# 32×32 images with one circle at a random position and radius and strong noise; the target is the circle's mask
def make(n, S=32):
    yy, xx = torch.meshgrid(torch.arange(S), torch.arange(S), indexing='ij')
    cx, cy, r = torch.randint(8, 24, (n,)), torch.randint(8, 24, (n,)), torch.randint(4, 8, (n,))
    mask = ((xx[None] - cx[:, None, None]) ** 2 + (yy[None] - cy[:, None, None]) ** 2 < r[:, None, None] ** 2).long()
    X = mask[:, None].float() + 0.4 * torch.randn(n, 1, S, S)
    return X, mask
X_train, y_train = make(384)
X_test, y_test = make(96)

def dice(pred, target):
    inter = ((pred == 1) & (target == 1)).sum().item()
    return 2 * inter / max(1, (pred == 1).sum().item() + (target == 1).sum().item())
model = SwinUNETR2D()

# 4) Loss function (cross-entropy + soft Dice, common in medical segmentation) and optimizer
def soft_dice_loss(logits, target):
    p = logits.softmax(1)[:, 1]; t = target.float()
    return 1 - (2 * (p * t).sum() + 1) / (p.sum() + t.sum() + 1)
opt = torch.optim.AdamW(model.parameters(), lr=2e-3, weight_decay=1e-4)

# 5) Training loop
EPOCHS, BATCH = 15, 16
for epoch in range(EPOCHS):
    model.train()
    perm, tot = torch.randperm(len(X_train)), 0.0
    for i in range(0, len(X_train), BATCH):
        idx = perm[i:i + BATCH]
        logits = model(X_train[idx])                                       # Forward Pass: B×2×H×W
        loss = F.cross_entropy(logits, y_train[idx]) + soft_dice_loss(logits, y_train[idx])
        opt.zero_grad(); loss.backward(); opt.step()
        tot += loss.item() * len(idx)
    if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
        print(f"epoch {epoch:3d}  loss={tot / len(X_train):.4f}")

# 6) Evaluation with the Dice coefficient on the test images
model.eval()
with torch.no_grad():
    pred = model(X_test).argmax(1)
print(f"test Dice = {dice(pred, y_test):.3f}")

# 7) Inference: the mask of a new image
with torch.no_grad():
    m = model(X_test[:1]).argmax(1)[0]
print("predicted foreground pixels:", int(m.sum()), " true:", int(y_test[0].sum()))

This code was run on 2026-11-20 in 15.8 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

test Dice = 0.988

Trainable parameters in the example: model = 105,890.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Advanced2021

Swin Transformer

Swin Transformer

A hierarchical vision transformer that computes attention within non-overlapping local windows and shifts the windows in successive layers to exchange information betwee…

Transformer Architectures

Advanced2022

UNETR

UNETR

A 3D segmentation model whose encoder is a pure ViT on 16³ volumetric patches; representations are taken from its layers (3, 6, 9, 12), reshaped and upsampled with trans…

Hybrid ArchitecturesTransformer Architectures

Intermediate2021

TransUNet

TransUNet

A medical segmentation model that uses a hybrid encoder: a CNN (ResNet-50) extracts a feature map, which is split into patches and passed through a ViT, followed by a ca…

Hybrid ArchitecturesTransformer Architectures

Intermediate2021

U-Net with Transformers

U-Net + Transformer (U-Net+Transformer)

A family of segmentation models that keeps the U-Net shape (an encoder–decoder with skip connections) and introduces the transformer somewhere: the bottleneck, the whole…

Hybrid ArchitecturesTransformer Architectures

Basic2015

U-Net

U-Net

A U-shaped segmentation architecture: a contracting path that captures context and an expanding path that restores spatial resolution, with skip connections that carry f…

FoundationsHybrid Architectures

Research ideas

  • Swin UNETR with self-supervised pretraining for segmenting multiple sclerosis lesions with limited local data.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    What is the advantage of the Swin encoder over the ViT in UNETR?

  2. Concept

    For which challenge was Swin UNETR developed?

  3. Concept

    What happens to the resolution after patch merging?

References

  1. Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H. R., & Xu, D. (2022). Swin UNETR: Swin transformers for semantic segmentation of brain tumors in MRI images. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries (BrainLes 2021), LNCS 12962, 272–284. https://doi.org/10.1007/978-3-031-08999-2_22

    DOIarXivAutomatically verified via Crossref – 2026-11-20

  2. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin Transformer: Hierarchical vision transformer using shifted windows. IEEE/CVF International Conference on Computer Vision (ICCV), 9992–10002. https://doi.org/10.1109/ICCV48922.2021.00986

    DOIarXivAutomatically verified via Crossref – 2026-11-20

  3. Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H. R., & Xu, D. (2022). UNETR: Transformers for 3D medical image segmentation. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 1748–1758. https://doi.org/10.1109/WACV51458.2022.00181

    DOIarXivAutomatically verified via Crossref – 2026-11-20

  4. Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention (MICCAI), LNCS 9351, 234–241. https://doi.org/10.1007/978-3-319-24574-4_28

    DOIarXivAutomatically verified via Crossref – 2026-11-20