Swin UNETR (Swin-UNETR)
A volumetric segmentation model whose encoder is a hierarchical Swin Transformer (shifted local windows and patch merging) that gives features at five scales, connected to a convolutional decoder through skip connections; it was developed for brain tumour segmentation in the BraTS 2021 challenge.
Overview
| Name | Swin UNETR |
|---|---|
| Year introduced | 2022 |
| Researchers | Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R. Roth, Daguang Xu |
| Institution | NVIDIA |
| Original paper | Swin UNETR: Swin transformers for semantic segmentation of brain tumors in MRI images DOI arXiv |
| Data type | Images |
| Domains | Medical image analysis |
| Computational cost | High, but linear in the number of tokens. |
| Data size | Hundreds of volumes (BraTS: more than 1000 training cases). |
Problem it solves
UNETR is single-scale and loses detail, and global attention on volumes is costly. A hierarchical windowed encoder combines efficiency with multiple scales.
Background and evolution
What came before: Swin Transformer, UNETR, U-Net
Core idea
The encoder applies 3D Swin blocks (shifted cubic windows) with patch merging between stages, producing representations at decreasing resolutions. Each representation passes through a residual convolutional block and is then concatenated in the decoder at the same scale, just like U-Net.
Interactive diagram
A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.
Select a layer in the diagram.
Components
| Component | Role |
|---|---|
| 3D patch partition | |
| Swin blocks (W-MSA / SW-MSA) | |
| Patch merging | |
| Residual conv blocks on skips | |
| CNN decoder |
Equations
1. Windowed attention cost (3D)
| $M$ | Window size |
| $hwd$ | Number of tokens |
2. Dice coefficient and soft Dice loss
| $P,G$ | Prediction and ground truth |
| $p_i$ | Foreground probability for pixel i |
How it works, step by step
- Split the volume into small patches.
- Apply the Swin stages with merging.
- Pass each scale through a convolutional block.
- Decode with concatenation.
- Train with Dice + CE.
Strengths and limitations
Strengths
- Multiple scales with wide context.
- The efficiency of windows.
- Strong results in BraTS 2021, according to the authors.
Limitations
- Implementation complexity.
- Large GPU memory for volumes.
When to use it, and when not?
Good for
- Volumetric segmentation that needs both context and detail.
Not suited for
- Small 2D data for which U-Net is enough.
Applications
| Domain | Use |
|---|---|
| Medical image analysis | Segmenting organs and tumours in CT and MRI. |
| Computer vision | Semantic segmentation of natural images. |
PyTorch code
A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.
The original version is 3D with 7³ windows and four stages; here is a 2D version with three stages and a 4×4 window that reuses the Swin blocks from the Swin page.
# A miniature 2D Swin UNETR: a hierarchical Swin encoder + a convolutional decoder
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
# 2) Define the model
def window_partition(x, M): # B×H×W×C → (B·nW)×M²×C
B, H, W, C = x.shape
x = x.view(B, H // M, M, W // M, M, C).permute(0, 1, 3, 2, 4, 5)
return x.reshape(-1, M * M, C)
def window_reverse(w, M, H, W): # (B·nW)×M²×C → B×H×W×C
B = w.shape[0] // ((H // M) * (W // M))
x = w.view(B, H // M, W // M, M, M, -1).permute(0, 1, 3, 2, 4, 5)
return x.reshape(B, H, W, -1)
class WindowAttention(nn.Module):
"""Attention inside an M×M window with a learned relative position bias (a (2M−1)² table per head)."""
def __init__(self, C, M, heads):
super().__init__()
self.h, self.dk, self.M = heads, C // heads, M
self.qkv, self.proj = nn.Linear(C, 3 * C), nn.Linear(C, C)
self.bias_table = nn.Parameter(torch.zeros((2 * M - 1) ** 2, heads))
coords = torch.stack(torch.meshgrid(torch.arange(M), torch.arange(M), indexing='ij')).flatten(1) # 2×M²
rel = (coords[:, :, None] - coords[:, None, :]).permute(1, 2, 0) + (M - 1) # M²×M²×2
self.register_buffer('rel_index', rel[..., 0] * (2 * M - 1) + rel[..., 1])
def forward(self, x, mask=None): # x: (B·nW)×M²×C ; mask: nW×M²×M²
Bw, N, C = x.shape
q, k, v = self.qkv(x).view(Bw, N, 3, self.h, self.dk).permute(2, 0, 3, 1, 4)
a = q @ k.transpose(-2, -1) / self.dk ** 0.5
a = a + self.bias_table[self.rel_index].permute(2, 0, 1)[None]
if mask is not None:
nW = mask.shape[0]
a = a.view(Bw // nW, nW, self.h, N, N) + mask[None, :, None]
a = a.view(Bw, self.h, N, N)
return self.proj((a.softmax(-1) @ v).transpose(1, 2).reshape(Bw, N, C))
class SwinBlock(nn.Module):
def __init__(self, C, H, W, heads, M=4, shift=0):
super().__init__()
self.H, self.W, self.M, self.s = H, W, M, shift
self.n1, self.attn, self.n2 = nn.LayerNorm(C), WindowAttention(C, M, heads), nn.LayerNorm(C)
self.mlp = nn.Sequential(nn.Linear(C, 2 * C), nn.GELU(), nn.Linear(2 * C, C))
mask = None
if shift: # a mask that prevents attention between regions that were not adjacent before the cyclic shift
img = torch.zeros(1, H, W, 1); cnt = 0
for hs in (slice(0, -M), slice(-M, -shift), slice(-shift, None)):
for ws in (slice(0, -M), slice(-M, -shift), slice(-shift, None)):
img[:, hs, ws, :] = cnt; cnt += 1
win = window_partition(img, M).squeeze(-1) # nW×M²
mask = (win[:, None, :] != win[:, :, None]).float() * -100.0
self.register_buffer('mask', mask)
def forward(self, x): # x: B×H×W×C
y = self.n1(x)
if self.s: y = torch.roll(y, (-self.s, -self.s), (1, 2)) # SW-MSA: cyclic shift
y = window_reverse(self.attn(window_partition(y, self.M), self.mask), self.M, self.H, self.W)
if self.s: y = torch.roll(y, (self.s, self.s), (1, 2))
x = x + y
return x + self.mlp(self.n2(x))
class PatchMerging(nn.Module):
"""Merge every 2×2 group of neighbouring patches: spatial resolution halves and channels double."""
def __init__(self, C):
super().__init__()
self.norm, self.red = nn.LayerNorm(4 * C), nn.Linear(4 * C, 2 * C, bias=False)
def forward(self, x):
x = torch.cat([x[:, 0::2, 0::2], x[:, 1::2, 0::2], x[:, 0::2, 1::2], x[:, 1::2, 1::2]], -1)
return self.red(self.norm(x))
def conv(ci, co):
return nn.Sequential(nn.Conv2d(ci, co, 3, padding=1), nn.InstanceNorm2d(co, affine=True), nn.LeakyReLU(0.01))
class SwinUNETR2D(nn.Module):
"""A hierarchical Swin encoder (regular then shifted windows + patch merging) feeding a convolutional decoder with skip connections at every scale."""
def __init__(self, k=2, C=16, M=4):
super().__init__()
self.embed = nn.Conv2d(1, C, 2, 2) # 32 → 16×16×C
self.s1 = nn.Sequential(SwinBlock(C, 16, 16, 2, M, 0), SwinBlock(C, 16, 16, 2, M, 2))
self.m1 = PatchMerging(C) # → 8×8×2C
self.s2 = nn.Sequential(SwinBlock(2 * C, 8, 8, 4, M, 0), SwinBlock(2 * C, 8, 8, 4, M, 2))
self.m2 = PatchMerging(2 * C) # → 4×4×4C
self.s3 = SwinBlock(4 * C, 4, 4, 4, M, 0)
self.enc0 = conv(1, C) # scale of the original image
self.up3, self.dec3 = nn.ConvTranspose2d(4 * C, 2 * C, 2, 2), conv(4 * C, 2 * C)
self.up2, self.dec2 = nn.ConvTranspose2d(2 * C, C, 2, 2), conv(2 * C, C)
self.up1, self.dec1 = nn.ConvTranspose2d(C, C, 2, 2), conv(2 * C, C)
self.out = nn.Conv2d(C, k, 1)
def forward(self, x):
h1 = self.s1(self.embed(x).permute(0, 2, 3, 1)) # B×16×16×C
h2 = self.s2(self.m1(h1)); h3 = self.s3(self.m2(h2))
t = lambda h: h.permute(0, 3, 1, 2)
d = self.dec3(torch.cat([self.up3(t(h3)), t(h2)], 1))
d = self.dec2(torch.cat([self.up2(d), t(h1)], 1))
d = self.dec1(torch.cat([self.up1(d), self.enc0(x)], 1))
return self.out(d)
# 3) Prepare the data
# 32×32 images with one circle at a random position and radius and strong noise; the target is the circle's mask
def make(n, S=32):
yy, xx = torch.meshgrid(torch.arange(S), torch.arange(S), indexing='ij')
cx, cy, r = torch.randint(8, 24, (n,)), torch.randint(8, 24, (n,)), torch.randint(4, 8, (n,))
mask = ((xx[None] - cx[:, None, None]) ** 2 + (yy[None] - cy[:, None, None]) ** 2 < r[:, None, None] ** 2).long()
X = mask[:, None].float() + 0.4 * torch.randn(n, 1, S, S)
return X, mask
X_train, y_train = make(384)
X_test, y_test = make(96)
def dice(pred, target):
inter = ((pred == 1) & (target == 1)).sum().item()
return 2 * inter / max(1, (pred == 1).sum().item() + (target == 1).sum().item())
model = SwinUNETR2D()
# 4) Loss function (cross-entropy + soft Dice, common in medical segmentation) and optimizer
def soft_dice_loss(logits, target):
p = logits.softmax(1)[:, 1]; t = target.float()
return 1 - (2 * (p * t).sum() + 1) / (p.sum() + t.sum() + 1)
opt = torch.optim.AdamW(model.parameters(), lr=2e-3, weight_decay=1e-4)
# 5) Training loop
EPOCHS, BATCH = 15, 16
for epoch in range(EPOCHS):
model.train()
perm, tot = torch.randperm(len(X_train)), 0.0
for i in range(0, len(X_train), BATCH):
idx = perm[i:i + BATCH]
logits = model(X_train[idx]) # Forward Pass: B×2×H×W
loss = F.cross_entropy(logits, y_train[idx]) + soft_dice_loss(logits, y_train[idx])
opt.zero_grad(); loss.backward(); opt.step()
tot += loss.item() * len(idx)
if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
print(f"epoch {epoch:3d} loss={tot / len(X_train):.4f}")
# 6) Evaluation with the Dice coefficient on the test images
model.eval()
with torch.no_grad():
pred = model(X_test).argmax(1)
print(f"test Dice = {dice(pred, y_test):.3f}")
# 7) Inference: the mask of a new image
with torch.no_grad():
m = model(X_test[:1]).argmax(1)[0]
print("predicted foreground pixels:", int(m.sum()), " true:", int(y_test[0].sum()))
This code was run on 2026-11-20 in 15.8 seconds (PyTorch (CPU), Python 3 – local build machine).
Sample output:
test Dice = 0.988
Trainable parameters in the example: model = 105,890.
These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.
Try it and go deeper
- Medical Image Analysis Lab Another section of the site
Related architectures
Swin Transformer
Swin Transformer
A hierarchical vision transformer that computes attention within non-overlapping local windows and shifts the windows in successive layers to exchange information betwee…
UNETR
UNETR
A 3D segmentation model whose encoder is a pure ViT on 16³ volumetric patches; representations are taken from its layers (3, 6, 9, 12), reshaped and upsampled with trans…
TransUNet
TransUNet
A medical segmentation model that uses a hybrid encoder: a CNN (ResNet-50) extracts a feature map, which is split into patches and passed through a ViT, followed by a ca…
U-Net with Transformers
U-Net + Transformer (U-Net+Transformer)
A family of segmentation models that keeps the U-Net shape (an encoder–decoder with skip connections) and introduces the transformer somewhere: the bottleneck, the whole…
U-Net
U-Net
A U-shaped segmentation architecture: a contracting path that captures context and an expanding path that restores spatial resolution, with skip connections that carry f…
No results match the filters.
Research ideas
- Swin UNETR with self-supervised pretraining for segmenting multiple sclerosis lesions with limited local data.
Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.
Test your understanding
What is the advantage of the Swin encoder over the ViT in UNETR?
Why? Windows and merging.
For which challenge was Swin UNETR developed?
Why? According to the paper's title.
What happens to the resolution after patch merging?
Why? As in Swin.
References
Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H. R., & Xu, D. (2022). Swin UNETR: Swin transformers for semantic segmentation of brain tumors in MRI images. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries (BrainLes 2021), LNCS 12962, 272–284. https://doi.org/10.1007/978-3-031-08999-2_22
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin Transformer: Hierarchical vision transformer using shifted windows. IEEE/CVF International Conference on Computer Vision (ICCV), 9992–10002. https://doi.org/10.1109/ICCV48922.2021.00986
Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H. R., & Xu, D. (2022). UNETR: Transformers for 3D medical image segmentation. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 1748–1758. https://doi.org/10.1109/WACV51458.2022.00181
Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention (MICCAI), LNCS 9351, 234–241. https://doi.org/10.1007/978-3-319-24574-4_28
