BasicFoundations

Residual Network (ResNet)

It adds skip connections that make each block learn the residual difference $F(x)$ instead of the full transformation, so training networks with hundreds of layers becomes possible.

Overview

NameResidual Network
Year introduced2016
ResearchersKaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
InstitutionMicrosoft Research
Original paperDeep residual learning for image recognition DOI arXiv
Data typeImages
DomainsComputer vision, Medical image analysis
Computational costDepends on depth; ResNet-50 is several times heavier than ResNet-18 in operations.
Data sizeWith transfer learning from ImageNet, small datasets are enough.

Problem it solves

Researchers noticed that increasing the depth of plain networks leads to higher training error (the degradation problem), not just to overfitting.

Background and evolution

What came before: Convolutional Neural Network, VGG

  1. 2015

    The first version was posted on arXiv and won the ILSVRC 2015 classification challenge. [ref]

  2. 2016

    The paper was published at CVPR 2016. [ref]

  3. 2017

    ResNeXt extended the idea with grouped parallel paths. [ref]

Core idea

If the required transformation is close to the identity, it is easier for the block to learn $F(x)\approx0$ than to learn the identity with nonlinear layers. So the block's output is $y=F(x)+x$. The connection passes the gradient directly to the early layers.

Example: In a 50-layer network, if some blocks are not useful, they can learn weights close to zero and pass the input through unchanged without harming performance.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
InputImage
StemA 7×7 convolution and initial pooling
Residual BlockTwo convolutions (or three in a bottleneck) with BatchNorm and ReLU
Skip ConnectionAdding the input to the output; a 1×1 projection when the shape changes
NormalizationBatchNorm
OutputGlobal Average Pooling, then Linear

Equations

1. The residual block

$$y=\mathcal{F}(x,\{W_i\})+x$$

Symbols
$x$Block input
$\mathcal{F}$The residual branch (convolutions)
$y$The output
Dimensions
Same shape as $x$
Numerical example
If $x=(1,2)$ and $\mathcal{F}(x)=(0.1,-0.3)$, then $y=(1.1,1.7)$.
Relation to the architecture
This addition is the essence of the architecture.

2. Projection when the shapes differ

$$y=\mathcal{F}(x)+W_s x$$

Symbols
$W_s$A 1×1 convolution to match channels or resolution
Dimensions
$W_s\in\mathbb{R}^{C_{out}\times C_{in}}$
Numerical example
From 64 channels to 128 with stride 2: $W_s$ is a 1×1 convolution with stride 2.
Relation to the architecture
It allows the addition when moving between stages.

3. Gradient flow

$$\frac{\partial \mathcal{L}}{\partial x}=\frac{\partial \mathcal{L}}{\partial y}\left(1+\frac{\partial \mathcal{F}}{\partial x}\right)$$

Symbols
$\partial\mathcal{L}/\partial y$The gradient coming from the next layer
Dimensions
Same shape as $x$
Numerical example
Even if $\partial\mathcal{F}/\partial x$ is very small, the 1 term remains, so the gradient flows.
Relation to the architecture
It explains why deep networks are easy to train.

How it works, step by step

  1. The image passes through the stem layer.
  2. Each stage processes several residual blocks.
  3. In each block $F(x)$ is computed and then $x$ is added to it.
  4. When moving between stages the channels are doubled and the resolution halved.
  5. Global Average Pooling is applied.
  6. The output is classified with a linear layer.

Strengths and limitations

Strengths

  • It allows very deep networks to be trained stably.
  • A standard backbone for many vision tasks.
  • Pretrained weights are widely available.

Limitations

  • Its deep variants are computationally heavy.
  • A local bias as in CNNs; global relationships need depth.

When to use it, and when not?

Good for

  • A backbone for classification, detection and segmentation.
  • A strong baseline for any computer vision study.

Not suited for

  • Very resource-constrained devices, where MobileNet or EfficientNet is preferred.

Applications

DomainUse
Computer visionImage classification and backbones for object detectors.
Medical image analysisClassifying radiology images and transfer learning on small medical data.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

Python (PyTorch)
# Residual Network (ResNet) — a simplified residual block
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
class BasicBlock(nn.Module):
    def __init__(self, c_in, c_out, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(c_in, c_out, 3, stride, 1, bias=False); self.bn1 = nn.BatchNorm2d(c_out)
        self.conv2 = nn.Conv2d(c_out, c_out, 3, 1, 1, bias=False);     self.bn2 = nn.BatchNorm2d(c_out)
        self.short = nn.Identity() if (stride == 1 and c_in == c_out) else nn.Sequential(
            nn.Conv2d(c_in, c_out, 1, stride, bias=False), nn.BatchNorm2d(c_out))   # projection W_s
    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return F.relu(out + self.short(x))       # y = F(x) + x

class TinyResNet(nn.Module):
    def __init__(self, n_classes=2):
        super().__init__()
        self.stem = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1, bias=False), nn.BatchNorm2d(16), nn.ReLU())
        self.layers = nn.Sequential(BasicBlock(16, 16), BasicBlock(16, 32, stride=2), BasicBlock(32, 32))
        self.head = nn.Linear(32, n_classes)
    def forward(self, x):
        x = self.layers(self.stem(x))
        return self.head(x.mean(dim=(2, 3)))     # Global Average Pooling

# 3) Prepare the data
def make(n):
    X = torch.randn(n, 1, 16, 16) * 0.3
    y = torch.randint(0, 2, (n,))
    pos = torch.randint(2, 14, (n,))
    for i in range(n):
        if y[i] == 0: X[i, 0, pos[i], 2:14] += 1.5
        else:         X[i, 0, 2:14, pos[i]] += 1.5
    return X, y
X_train, y_train = make(512)
X_test, y_test = make(128)
model = TinyResNet()

# 4) Loss function and optimizer
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=5e-3)

# 5) Training loop with the forward pass
EPOCHS, BATCH = 5, 32
for epoch in range(EPOCHS):
    model.train()
    perm = torch.randperm(len(X_train))
    total = 0.0
    for i in range(0, len(X_train), BATCH):
        idx = perm[i:i + BATCH]
        xb, yb = X_train[idx], y_train[idx]
        out = model(xb)                       # Forward Pass
        loss = loss_fn(out, yb)
        opt.zero_grad()
        loss.backward()                    # Backpropagation
        opt.step()
        total += loss.item() * len(idx)
    if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
        print(f"epoch {epoch:3d}  loss={total / len(X_train):.4f}")

# 6) Evaluation
model.eval()
with torch.no_grad():
    xb = X_test
    out = model(xb)
    acc = (out.argmax(dim=-1) == y_test).float().mean().item()
    print(f"test accuracy = {acc:.3f}")

# 7) Inference on a new sample
with torch.no_grad():
    xb = X_test[:1]
    out = model(xb)
    print("predicted class:", out.argmax(dim=-1).item())

This code was run on 2026-11-20 in 3.3 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

test accuracy = 1.000

Trainable parameters in the example: model = 38,002.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Beginner1998

Convolutional Neural Network

Convolutional Neural Network (CNN)

A network that uses small convolution filters with shared weights sliding over the image, so it captures local patterns with far fewer parameters than an MLP.

Foundations

Basic2015

Multi-Branch Neural Networks

Multi-Branch Neural Networks

Architectures that process the same input or different inputs through parallel branches (different kernel sizes, different modalities or different scales) and then merge…

Hybrid Architectures

Advanced2019

Deep Equilibrium Models

Deep Equilibrium Models (DEQ)

Instead of stacking L layers, the model looks for a fixed point of a single layer z^*=f_θ(z^*,x), as if it were an infinitely deep network with shared weights, and compu…

Advanced Neural Architectures

Basic2015

U-Net

U-Net

A U-shaped segmentation architecture: a contracting path that captures context and an expanding path that restores spatial resolution, with skip connections that carry f…

FoundationsHybrid Architectures

Research ideas

  • Study the effect of skip connections on interpretability with Grad-CAM maps in medical images.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    What does the residual block in ResNet learn?

  2. Concept

    Which problem did ResNet mainly address?

  3. Concept

    When do we need a 1×1 convolution in the skip connection?

References

  1. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778. https://doi.org/10.1109/CVPR.2016.90

    DOIarXivAutomatically verified via Crossref – 2026-11-20

  2. Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning (ICML), PMLR 37, 448–456. https://arxiv.org/abs/1502.03167

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  3. Xie, S., Girshick, R., Dollár, P., Tu, Z., & He, K. (2017). Aggregated residual transformations for deep neural networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 5987–5995. https://doi.org/10.1109/CVPR.2017.634

    DOIarXivAutomatically verified via Crossref – 2026-11-20