Residual Network (ResNet)
It adds skip connections that make each block learn the residual difference $F(x)$ instead of the full transformation, so training networks with hundreds of layers becomes possible.
Overview
| Name | Residual Network |
|---|---|
| Year introduced | 2016 |
| Researchers | Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun |
| Institution | Microsoft Research |
| Original paper | Deep residual learning for image recognition DOI arXiv |
| Data type | Images |
| Domains | Computer vision, Medical image analysis |
| Computational cost | Depends on depth; ResNet-50 is several times heavier than ResNet-18 in operations. |
| Data size | With transfer learning from ImageNet, small datasets are enough. |
Problem it solves
Researchers noticed that increasing the depth of plain networks leads to higher training error (the degradation problem), not just to overfitting.
Background and evolution
What came before: Convolutional Neural Network, VGG
Core idea
If the required transformation is close to the identity, it is easier for the block to learn $F(x)\approx0$ than to learn the identity with nonlinear layers. So the block's output is $y=F(x)+x$. The connection passes the gradient directly to the early layers.
Interactive diagram
A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.
Select a layer in the diagram.
Components
| Component | Role |
|---|---|
| Input | Image |
| Stem | A 7×7 convolution and initial pooling |
| Residual Block | Two convolutions (or three in a bottleneck) with BatchNorm and ReLU |
| Skip Connection | Adding the input to the output; a 1×1 projection when the shape changes |
| Normalization | BatchNorm |
| Output | Global Average Pooling, then Linear |
Equations
1. The residual block
| $x$ | Block input |
| $\mathcal{F}$ | The residual branch (convolutions) |
| $y$ | The output |
2. Projection when the shapes differ
| $W_s$ | A 1×1 convolution to match channels or resolution |
3. Gradient flow
| $\partial\mathcal{L}/\partial y$ | The gradient coming from the next layer |
How it works, step by step
- The image passes through the stem layer.
- Each stage processes several residual blocks.
- In each block $F(x)$ is computed and then $x$ is added to it.
- When moving between stages the channels are doubled and the resolution halved.
- Global Average Pooling is applied.
- The output is classified with a linear layer.
Strengths and limitations
Strengths
- It allows very deep networks to be trained stably.
- A standard backbone for many vision tasks.
- Pretrained weights are widely available.
Limitations
- Its deep variants are computationally heavy.
- A local bias as in CNNs; global relationships need depth.
When to use it, and when not?
Good for
- A backbone for classification, detection and segmentation.
- A strong baseline for any computer vision study.
Not suited for
- Very resource-constrained devices, where MobileNet or EfficientNet is preferred.
Applications
| Domain | Use |
|---|---|
| Computer vision | Image classification and backbones for object detectors. |
| Medical image analysis | Classifying radiology images and transfer learning on small medical data. |
PyTorch code
A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.
# Residual Network (ResNet) — a simplified residual block
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
# 2) Define the model
class BasicBlock(nn.Module):
def __init__(self, c_in, c_out, stride=1):
super().__init__()
self.conv1 = nn.Conv2d(c_in, c_out, 3, stride, 1, bias=False); self.bn1 = nn.BatchNorm2d(c_out)
self.conv2 = nn.Conv2d(c_out, c_out, 3, 1, 1, bias=False); self.bn2 = nn.BatchNorm2d(c_out)
self.short = nn.Identity() if (stride == 1 and c_in == c_out) else nn.Sequential(
nn.Conv2d(c_in, c_out, 1, stride, bias=False), nn.BatchNorm2d(c_out)) # projection W_s
def forward(self, x):
out = F.relu(self.bn1(self.conv1(x)))
out = self.bn2(self.conv2(out))
return F.relu(out + self.short(x)) # y = F(x) + x
class TinyResNet(nn.Module):
def __init__(self, n_classes=2):
super().__init__()
self.stem = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1, bias=False), nn.BatchNorm2d(16), nn.ReLU())
self.layers = nn.Sequential(BasicBlock(16, 16), BasicBlock(16, 32, stride=2), BasicBlock(32, 32))
self.head = nn.Linear(32, n_classes)
def forward(self, x):
x = self.layers(self.stem(x))
return self.head(x.mean(dim=(2, 3))) # Global Average Pooling
# 3) Prepare the data
def make(n):
X = torch.randn(n, 1, 16, 16) * 0.3
y = torch.randint(0, 2, (n,))
pos = torch.randint(2, 14, (n,))
for i in range(n):
if y[i] == 0: X[i, 0, pos[i], 2:14] += 1.5
else: X[i, 0, 2:14, pos[i]] += 1.5
return X, y
X_train, y_train = make(512)
X_test, y_test = make(128)
model = TinyResNet()
# 4) Loss function and optimizer
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=5e-3)
# 5) Training loop with the forward pass
EPOCHS, BATCH = 5, 32
for epoch in range(EPOCHS):
model.train()
perm = torch.randperm(len(X_train))
total = 0.0
for i in range(0, len(X_train), BATCH):
idx = perm[i:i + BATCH]
xb, yb = X_train[idx], y_train[idx]
out = model(xb) # Forward Pass
loss = loss_fn(out, yb)
opt.zero_grad()
loss.backward() # Backpropagation
opt.step()
total += loss.item() * len(idx)
if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
print(f"epoch {epoch:3d} loss={total / len(X_train):.4f}")
# 6) Evaluation
model.eval()
with torch.no_grad():
xb = X_test
out = model(xb)
acc = (out.argmax(dim=-1) == y_test).float().mean().item()
print(f"test accuracy = {acc:.3f}")
# 7) Inference on a new sample
with torch.no_grad():
xb = X_test[:1]
out = model(xb)
print("predicted class:", out.argmax(dim=-1).item())
This code was run on 2026-11-20 in 3.3 seconds (PyTorch (CPU), Python 3 – local build machine).
Sample output:
test accuracy = 1.000
Trainable parameters in the example: model = 38,002.
These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.
Try it and go deeper
- Neural Architecture Designer
- AI Model Atlas Another section of the site
Related architectures
Convolutional Neural Network
Convolutional Neural Network (CNN)
A network that uses small convolution filters with shared weights sliding over the image, so it captures local patterns with far fewer parameters than an MLP.
Multi-Branch Neural Networks
Multi-Branch Neural Networks
Architectures that process the same input or different inputs through parallel branches (different kernel sizes, different modalities or different scales) and then merge…
Deep Equilibrium Models
Deep Equilibrium Models (DEQ)
Instead of stacking L layers, the model looks for a fixed point of a single layer z^*=f_θ(z^*,x), as if it were an infinitely deep network with shared weights, and compu…
U-Net
U-Net
A U-shaped segmentation architecture: a contracting path that captures context and an expanding path that restores spatial resolution, with skip connections that carry f…
No results match the filters.
Research ideas
- Study the effect of skip connections on interpretability with Grad-CAM maps in medical images.
Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.
Test your understanding
What does the residual block in ResNet learn?
Why? The output is $y=F(x)+x$, so the block learns what must be added to the input.
Which problem did ResNet mainly address?
Why? The authors observed that deeper networks without connections give higher training error, and this is not overfitting.
When do we need a 1×1 convolution in the skip connection?
Why? The addition requires matching shapes.
References
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778. https://doi.org/10.1109/CVPR.2016.90
Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning (ICML), PMLR 37, 448–456. https://arxiv.org/abs/1502.03167
Xie, S., Girshick, R., Dollár, P., Tu, Z., & He, K. (2017). Aggregated residual transformations for deep neural networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 5987–5995. https://doi.org/10.1109/CVPR.2017.634
