BasicHybrid Architectures

Attention U-Net

A U-Net with attention gates on the skip connections that learn to highlight the relevant regions and suppress the background before the features are merged.

Overview

NameAttention U-Net
Year introduced2018
ResearchersOzan Oktay, Jo Schlemper, Loic Le Folgoc et al.
Original paperAttention U-Net: Learning where to look for the pancreas arXiv
Data typeImages
DomainsMedical image analysis

Problem it solves

Small organs such as the pancreas occupy a small part of the image, so the skip connections carry many unhelpful background features.

Background and evolution

What came before: U-Net

  1. 2018

    It was introduced at MIDL 2018 and tested on pancreas segmentation in CT images. [ref]

Core idea

Before the encoder map $x$ is concatenated with the decoder, a weight map $\alpha\in[0,1]$ is computed for each pixel using a gating signal $g$ from the deeper level. $x$ is multiplied by $\alpha$ so that only the important regions pass.

Example: When segmenting the pancreas, the gate learns to give values close to 1 around the pancreas and values close to 0 in the liver and intestines.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
Encoder/DecoderAs in U-Net
Gating Signal gFeatures of the deeper level
Attention GateAdditive fusion, then Sigmoid
Skip ConnectionConcatenating the features after multiplying by the weights

Equations

1. Attention gate

$$\alpha_i=\sigma_2\!\left(\psi^{\top}\sigma_1\!\left(W_x^{\top}x_i+W_g^{\top}g_i+b_g\right)+b_\psi\right),\qquad \hat{x}_i=\alpha_i\,x_i$$

Symbols
$x_i$Encoder features at pixel i
$g_i$Gating signal
$\sigma_1$ReLU
$\sigma_2$Sigmoid
Dimensions
$\alpha_i\in[0,1]$ for each pixel
Numerical example
If the pre-Sigmoid value is $2$ then $\alpha\approx0.88$, and if it is $-2$ then $\alpha\approx0.12$.
Relation to the architecture
It sets how much of each pixel passes through the skip connection.

How it works, step by step

  1. The input passes through the U-Net encoder.
  2. At each decoder level the gating signal is computed.
  3. The α map is computed from x and g.
  4. The skip features are multiplied by α.
  5. They are concatenated with the decoder and upsampling continues.

Strengths and limitations

Strengths

  • Improves the segmentation of small targets.
  • A light addition to U-Net.
  • Gives attention maps that can be displayed.

Limitations

  • Increases the cost slightly.
  • The benefit depends on the task and is not guaranteed.

When to use it, and when not?

Good for

  • Segmenting small organs or lesions in a large background.

Not suited for

  • When the targets occupy most of the image.

Applications

DomainUse
Medical image analysisSegmenting the pancreas and small lesions in CT and MRI.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

Python (PyTorch)
# A miniature Attention U-Net
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
def block(ci, co):
    return nn.Sequential(nn.Conv2d(ci, co, 3, padding=1), nn.BatchNorm2d(co), nn.ReLU(),
                         nn.Conv2d(co, co, 3, padding=1), nn.BatchNorm2d(co), nn.ReLU())
class AttentionGate(nn.Module):
    def __init__(self, cx, cg, ci):
        super().__init__()
        self.wx, self.wg = nn.Conv2d(cx, ci, 1), nn.Conv2d(cg, ci, 1)
        self.psi = nn.Conv2d(ci, 1, 1)
    def forward(self, x, g):
        a = torch.sigmoid(self.psi(F.relu(self.wx(x) + self.wg(g))))   # B×1×H×W
        return x * a
class AttUNet(nn.Module):
    def __init__(self, c=1, k=2):
        super().__init__()
        self.e1, self.e2, self.mid = block(c, 8), block(8, 16), block(16, 32)
        self.up2, self.ag2, self.d2 = nn.ConvTranspose2d(32, 16, 2, 2), AttentionGate(16, 16, 8), block(32, 16)
        self.up1, self.ag1, self.d1 = nn.ConvTranspose2d(16, 8, 2, 2), AttentionGate(8, 8, 4), block(16, 8)
        self.out = nn.Conv2d(8, k, 1)
    def forward(self, x):
        e1 = self.e1(x); e2 = self.e2(F.max_pool2d(e1, 2)); m = self.mid(F.max_pool2d(e2, 2))
        u2 = self.up2(m);  d2 = self.d2(torch.cat([u2, self.ag2(e2, u2)], 1))
        u1 = self.up1(d2); d1 = self.d1(torch.cat([u1, self.ag1(e1, u1)], 1))
        return self.out(d1)

# 3) Prepare the data
# Small targets (circles with a radius of 2–3) in a background with distracting squares
def make(n, S=32):
    yy, xx = torch.meshgrid(torch.arange(S), torch.arange(S), indexing='ij')
    cx, cy, r = torch.randint(6, 26, (n,)), torch.randint(6, 26, (n,)), torch.randint(2, 4, (n,))
    mask = ((xx[None] - cx[:, None, None]) ** 2 + (yy[None] - cy[:, None, None]) ** 2 <= r[:, None, None] ** 2).long()
    X = mask[:, None].float() * 1.5 + 0.4 * torch.randn(n, 1, S, S)
    for i in range(n):
        a, b = torch.randint(0, S - 6, (2,)).tolist()
        X[i, 0, a:a + 6, b:b + 6] += 0.6       # distractor: a fainter square
    return X, mask
X_train, y_train = make(256)
X_test, y_test = make(64)
model = AttUNet()

# 4) The loss: weighted CE because the target is small
loss_fn = nn.CrossEntropyLoss(weight=torch.tensor([1.0, 5.0]))
opt = torch.optim.Adam(model.parameters(), lr=1e-2)

# 5) Training loop
for epoch in range(8):
    model.train()
    for i in range(0, len(X_train), 32):
        xb, yb = X_train[i:i + 32], y_train[i:i + 32]
        loss = loss_fn(model(xb), yb)
        opt.zero_grad(); loss.backward(); opt.step()
    print(f"epoch {epoch}  loss={loss.item():.4f}")

# 6) Evaluation with the Dice coefficient
model.eval()
with torch.no_grad():
    pred = model(X_test).argmax(1)
inter = ((pred == 1) & (y_test == 1)).sum().item()
print(f"test Dice = {2 * inter / max(1, (pred == 1).sum().item() + (y_test == 1).sum().item()):.3f}")

# 7) Inference
with torch.no_grad():
    print("foreground pixels in first test image:", int(model(X_test[:1]).argmax(1).sum()))

This code was run on 2026-11-20 in 5.5 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

test Dice = 0.974

Trainable parameters in the example: model = 30,008.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Basic2015

U-Net

U-Net

A U-shaped segmentation architecture: a contracting path that captures context and an expanding path that restores spatial resolution, with skip connections that carry f…

FoundationsHybrid Architectures

Intermediate2021

TransUNet

TransUNet

A medical segmentation model that uses a hybrid encoder: a CNN (ResNet-50) extracts a feature map, which is split into patches and passed through a ViT, followed by a ca…

Hybrid ArchitecturesTransformer Architectures

Basic2018

Attention-Augmented Convolutional Networks

Attention-based CNN (Attention CNN)

Convolutional networks that add lightweight attention modules that reweight the channels (which features matter most) or the positions (where the model looks), at a very…

Hybrid Architectures

Research ideas

  • Evaluate attention gates against Swin-UNETR for segmenting small lesions, with a sensitivity analysis by lesion size.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    Where are the attention gates placed in Attention U-Net?

  2. Concept

    What is the range of the attention coefficients α in the gate?

  3. Concept

    For which kind of task do attention gates help most?

References

  1. Oktay, O., Schlemper, J., Le Folgoc, L., Lee, M., Heinrich, M., Misawa, K., et al. (2018). Attention U-Net: Learning where to look for the pancreas. Medical Imaging with Deep Learning (MIDL). https://arxiv.org/abs/1804.03999

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  2. Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention (MICCAI), LNCS 9351, 234–241. https://doi.org/10.1007/978-3-319-24574-4_28

    DOIarXivAutomatically verified via Crossref – 2026-11-20