BasicHybrid Architectures

Multi-Branch Neural Networks (Multi-Branch)

Architectures that process the same input or different inputs through parallel branches (different kernel sizes, different modalities or different scales) and then merge the results by concatenation or addition.

Overview

NameMulti-Branch Neural Networks
Year introduced2015
ResearchersReference example: Christian Szegedy and colleagues (Inception/GoogLeNet); also Simonyan and Zisserman (two-stream networks, 2014)
InstitutionGoogle
Original paperGoing deeper with convolutions DOI arXiv
Data typeImages, Sequences and time series, Tabular data
DomainsComputer vision, Audio and speech, Medical image analysis
Computational costThe sum of the branches' costs.
Data sizeSmall to large.

Problem it solves

Choosing a single kernel size or representation may not suit every pattern; and multi-source data needs specialised processing for each source before merging.

Background and evolution

What came before: Convolutional Neural Network

  1. 2014

    Two-stream networks for video: a branch for appearance and a branch for optical flow. [ref]

  2. 2015

    Inception/GoogLeNet: 1×1, 3×3 and 5×5 branches and pooling (won ILSVRC 2014). [ref]

  3. 2017

    ResNeXt: many homogeneous branches (cardinality) with a residual connection. [ref]

Core idea

In Inception each branch sees the input at a different scale, and the large convolutions are preceded by a 1×1 convolution that reduces the channels so the cost stays reasonable. In multi-input networks each branch specialises in a modality (an image, clinical data) and the features are then merged.

Example: A diagnosis that combines a branch for the radiology image with a branch for tabular lab data.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
Parallel branchesDifferent kernels or modalities
1×1 bottleneckChannel reduction
FusionConcatenation or addition
Optional residualAs in ResNeXt

Equations

1. Merging by concatenation

$$y=\big[f_1(x)\,\|\,f_2(x)\,\|\,\dots\,\|\,f_K(x)\big],\qquad C_{out}=\sum_kC_k$$

Symbols
$f_k$Branch k
$\|$Channel concatenation
Dimensions
$y\in\mathbb{R}^{C_{out}\times H\times W}$
Numerical example
Branches with 64, 128, 32 and 32 channels ⇒ $C_{out}=256$ (the inception 3a block in GoogLeNet).
Relation to the architecture
Each branch adds a kind of feature.

2. Savings from a 1×1 convolution

$$\#\text{params}_{direct}=k^2C_{in}C_{out}\quad\text{vs}\quad C_{in}C_r+k^2C_rC_{out}$$

Symbols
$C_r$Reduction channels
Dimensions
Parameter count (without biases)
Numerical example
$C_{in}=192$, $k=5$, $C_{out}=32$: directly $153{,}600$, and with $C_r=16$: $3{,}072+12{,}800=15{,}872$ (about 10 times less).
Relation to the architecture
It makes large branches feasible.

How it works, step by step

  1. Pass the input to the branches.
  2. Apply each branch's processing.
  3. Merge.
  4. Continue with the next blocks or the head.

Strengths and limitations

Strengths

  • Multi-scale features.
  • Natural fusion of modalities.
  • Efficiency with 1×1 bottlenecks.

Limitations

  • Complex manual design.
  • Branches may learn at different speeds.
  • More memory.

When to use it, and when not?

Good for

  • Patterns at multiple scales.
  • Multimodal data.

Not suited for

  • Simple tasks for which a chain of layers is enough.

Applications

DomainUse
Computer visionClassification (Inception).
Medical image analysisFusing the image with clinical data.
Audio and speechFusing spectral and temporal representations.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

Python (PyTorch)
# Multi-branch Inception: patterns at different scales
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
class InceptionBlock(nn.Module):
    """A multi-branch block: convolutions of different sizes on the same input in parallel, then channel concatenation."""
    def __init__(self, ci, c1=8, c3=8, c5=8, cp=8):
        super().__init__()
        self.b1 = nn.Sequential(nn.Conv2d(ci, c1, 1), nn.ReLU())
        self.b3 = nn.Sequential(nn.Conv2d(ci, 8, 1), nn.ReLU(), nn.Conv2d(8, c3, 3, padding=1), nn.ReLU())   # 1×1 to reduce the channels first
        self.b5 = nn.Sequential(nn.Conv2d(ci, 8, 1), nn.ReLU(), nn.Conv2d(8, c5, 5, padding=2), nn.ReLU())
        self.bp = nn.Sequential(nn.MaxPool2d(3, 1, 1), nn.Conv2d(ci, cp, 1), nn.ReLU())
    def forward(self, x):
        return torch.cat([self.b1(x), self.b3(x), self.b5(x), self.bp(x)], 1)

class MultiBranchNet(nn.Module):
    def __init__(self, n_classes=3):
        super().__init__()
        self.stem = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1), nn.ReLU())
        self.inc1 = InceptionBlock(16)                      # → 32 channels
        self.inc2 = nn.Sequential(nn.MaxPool2d(2), InceptionBlock(32))
        self.head = nn.Linear(32, n_classes)
    def forward(self, x):
        return self.head(self.inc2(self.inc1(self.stem(x))).mean((2, 3)))

# 3) Prepare the data
# Classes that differ in scale: a small dot (0), a medium blob (1), a large ring (2) — branches with different kernel sizes help here
def make(n, S=16):
    yy, xx = torch.meshgrid(torch.arange(S).float(), torch.arange(S).float(), indexing='ij')
    y = torch.randint(0, 3, (n,)); X = torch.randn(n, 1, S, S) * 0.3
    for i in range(n):
        cx, cy = torch.randint(5, 11, (2,)).float()
        d = ((xx - cx) ** 2 + (yy - cy) ** 2).sqrt()
        if y[i] == 0:   X[i, 0] += 1.5 * (d < 1.2)
        elif y[i] == 1: X[i, 0] += 1.5 * (d < 2.8)
        else:           X[i, 0] += 1.5 * ((d > 3.5) & (d < 4.8))
    return X, y
X_train, y_train = make(900)
X_test, y_test = make(300)
model = MultiBranchNet()

# 4) Loss function and optimizer
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=5e-3)

# 5) Training loop with the forward pass
EPOCHS, BATCH = 10, 32
for epoch in range(EPOCHS):
    model.train()
    perm = torch.randperm(len(X_train))
    total = 0.0
    for i in range(0, len(X_train), BATCH):
        idx = perm[i:i + BATCH]
        xb, yb = X_train[idx], y_train[idx]
        out = model(xb)                       # Forward Pass
        loss = loss_fn(out, yb)
        opt.zero_grad()
        loss.backward()                    # Backpropagation
        opt.step()
        total += loss.item() * len(idx)
    if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
        print(f"epoch {epoch:3d}  loss={total / len(X_train):.4f}")

# 6) Evaluation
model.eval()
with torch.no_grad():
    xb = X_test
    out = model(xb)
    acc = (out.argmax(dim=-1) == y_test).float().mean().item()
    print(f"test accuracy = {acc:.3f}")

# 7) Inference on a new sample
with torch.no_grad():
    xb = X_test[:1]
    out = model(xb)
    print("predicted class:", out.argmax(dim=-1).item())

This code was run on 2026-11-20 in 6.9 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

test accuracy = 1.000

Trainable parameters in the example: model = 6,243.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Beginner1998

Convolutional Neural Network

Convolutional Neural Network (CNN)

A network that uses small convolution filters with shared weights sliding over the image, so it captures local patterns with far fewer parameters than an MLP.

Foundations

Basic2016

Residual Network

Residual Network (ResNet)

It adds skip connections that make each block learn the residual difference F(x) instead of the full transformation, so training networks with hundreds of layers becomes…

Foundations

Intermediate2019

Multimodal transformers

Multimodal Transformers (MM-Transformer)

Transformers that process more than one modality (image, text, audio...) either in a single sequence with a modality-type embedding (single-stream), or in two streams th…

Transformer Architectures

Intermediate1991

Mixture of experts

Mixture of Experts (MoE)

Several "expert" networks and a gating network that decides, for each input, the weight of each expert in the output. The experts learn to specialize in different region…

Specialized Architectures

Research ideas

  • A multi-branch network that fuses mammogram images with clinical data, with an analysis of each branch's contribution.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    What is the benefit of a 1×1 convolution before the 5×5 in Inception?

  2. Concept

    Four branches with 16, 32, 8 and 8 channels: how many output channels after concatenation?

  3. Concept

    What are the two branches in a two-stream video network?

References

  1. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1–9. https://doi.org/10.1109/CVPR.2015.7298594

    DOIarXivAutomatically verified via Crossref – 2026-11-20

  2. Simonyan, K., & Zisserman, A. (2014). Two-stream convolutional networks for action recognition in videos. Advances in Neural Information Processing Systems (NeurIPS) 27. https://arxiv.org/abs/1406.2199

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  3. Xie, S., Girshick, R., Dollár, P., Tu, Z., & He, K. (2017). Aggregated residual transformations for deep neural networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 5987–5995. https://doi.org/10.1109/CVPR.2017.634

    DOIarXivAutomatically verified via Crossref – 2026-11-20