Multi-Branch Neural Networks (Multi-Branch)
Architectures that process the same input or different inputs through parallel branches (different kernel sizes, different modalities or different scales) and then merge the results by concatenation or addition.
Overview
| Name | Multi-Branch Neural Networks |
|---|---|
| Year introduced | 2015 |
| Researchers | Reference example: Christian Szegedy and colleagues (Inception/GoogLeNet); also Simonyan and Zisserman (two-stream networks, 2014) |
| Institution | |
| Original paper | Going deeper with convolutions DOI arXiv |
| Data type | Images, Sequences and time series, Tabular data |
| Domains | Computer vision, Audio and speech, Medical image analysis |
| Computational cost | The sum of the branches' costs. |
| Data size | Small to large. |
Problem it solves
Choosing a single kernel size or representation may not suit every pattern; and multi-source data needs specialised processing for each source before merging.
Background and evolution
What came before: Convolutional Neural Network
Core idea
In Inception each branch sees the input at a different scale, and the large convolutions are preceded by a 1×1 convolution that reduces the channels so the cost stays reasonable. In multi-input networks each branch specialises in a modality (an image, clinical data) and the features are then merged.
Interactive diagram
A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.
Select a layer in the diagram.
Components
| Component | Role |
|---|---|
| Parallel branches | Different kernels or modalities |
| 1×1 bottleneck | Channel reduction |
| Fusion | Concatenation or addition |
| Optional residual | As in ResNeXt |
Equations
1. Merging by concatenation
| $f_k$ | Branch k |
| $\|$ | Channel concatenation |
2. Savings from a 1×1 convolution
| $C_r$ | Reduction channels |
How it works, step by step
- Pass the input to the branches.
- Apply each branch's processing.
- Merge.
- Continue with the next blocks or the head.
Strengths and limitations
Strengths
- Multi-scale features.
- Natural fusion of modalities.
- Efficiency with 1×1 bottlenecks.
Limitations
- Complex manual design.
- Branches may learn at different speeds.
- More memory.
When to use it, and when not?
Good for
- Patterns at multiple scales.
- Multimodal data.
Not suited for
- Simple tasks for which a chain of layers is enough.
Applications
| Domain | Use |
|---|---|
| Computer vision | Classification (Inception). |
| Medical image analysis | Fusing the image with clinical data. |
| Audio and speech | Fusing spectral and temporal representations. |
PyTorch code
A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.
# Multi-branch Inception: patterns at different scales
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
# 2) Define the model
class InceptionBlock(nn.Module):
"""A multi-branch block: convolutions of different sizes on the same input in parallel, then channel concatenation."""
def __init__(self, ci, c1=8, c3=8, c5=8, cp=8):
super().__init__()
self.b1 = nn.Sequential(nn.Conv2d(ci, c1, 1), nn.ReLU())
self.b3 = nn.Sequential(nn.Conv2d(ci, 8, 1), nn.ReLU(), nn.Conv2d(8, c3, 3, padding=1), nn.ReLU()) # 1×1 to reduce the channels first
self.b5 = nn.Sequential(nn.Conv2d(ci, 8, 1), nn.ReLU(), nn.Conv2d(8, c5, 5, padding=2), nn.ReLU())
self.bp = nn.Sequential(nn.MaxPool2d(3, 1, 1), nn.Conv2d(ci, cp, 1), nn.ReLU())
def forward(self, x):
return torch.cat([self.b1(x), self.b3(x), self.b5(x), self.bp(x)], 1)
class MultiBranchNet(nn.Module):
def __init__(self, n_classes=3):
super().__init__()
self.stem = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1), nn.ReLU())
self.inc1 = InceptionBlock(16) # → 32 channels
self.inc2 = nn.Sequential(nn.MaxPool2d(2), InceptionBlock(32))
self.head = nn.Linear(32, n_classes)
def forward(self, x):
return self.head(self.inc2(self.inc1(self.stem(x))).mean((2, 3)))
# 3) Prepare the data
# Classes that differ in scale: a small dot (0), a medium blob (1), a large ring (2) — branches with different kernel sizes help here
def make(n, S=16):
yy, xx = torch.meshgrid(torch.arange(S).float(), torch.arange(S).float(), indexing='ij')
y = torch.randint(0, 3, (n,)); X = torch.randn(n, 1, S, S) * 0.3
for i in range(n):
cx, cy = torch.randint(5, 11, (2,)).float()
d = ((xx - cx) ** 2 + (yy - cy) ** 2).sqrt()
if y[i] == 0: X[i, 0] += 1.5 * (d < 1.2)
elif y[i] == 1: X[i, 0] += 1.5 * (d < 2.8)
else: X[i, 0] += 1.5 * ((d > 3.5) & (d < 4.8))
return X, y
X_train, y_train = make(900)
X_test, y_test = make(300)
model = MultiBranchNet()
# 4) Loss function and optimizer
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=5e-3)
# 5) Training loop with the forward pass
EPOCHS, BATCH = 10, 32
for epoch in range(EPOCHS):
model.train()
perm = torch.randperm(len(X_train))
total = 0.0
for i in range(0, len(X_train), BATCH):
idx = perm[i:i + BATCH]
xb, yb = X_train[idx], y_train[idx]
out = model(xb) # Forward Pass
loss = loss_fn(out, yb)
opt.zero_grad()
loss.backward() # Backpropagation
opt.step()
total += loss.item() * len(idx)
if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
print(f"epoch {epoch:3d} loss={total / len(X_train):.4f}")
# 6) Evaluation
model.eval()
with torch.no_grad():
xb = X_test
out = model(xb)
acc = (out.argmax(dim=-1) == y_test).float().mean().item()
print(f"test accuracy = {acc:.3f}")
# 7) Inference on a new sample
with torch.no_grad():
xb = X_test[:1]
out = model(xb)
print("predicted class:", out.argmax(dim=-1).item())
This code was run on 2026-11-20 in 6.9 seconds (PyTorch (CPU), Python 3 – local build machine).
Sample output:
test accuracy = 1.000
Trainable parameters in the example: model = 6,243.
These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.
Try it and go deeper
Related architectures
Convolutional Neural Network
Convolutional Neural Network (CNN)
A network that uses small convolution filters with shared weights sliding over the image, so it captures local patterns with far fewer parameters than an MLP.
Residual Network
Residual Network (ResNet)
It adds skip connections that make each block learn the residual difference F(x) instead of the full transformation, so training networks with hundreds of layers becomes…
Multimodal transformers
Multimodal Transformers (MM-Transformer)
Transformers that process more than one modality (image, text, audio...) either in a single sequence with a modality-type embedding (single-stream), or in two streams th…
Mixture of experts
Mixture of Experts (MoE)
Several "expert" networks and a gating network that decides, for each input, the weight of each expert in the output. The experts learn to specialize in different region…
No results match the filters.
Research ideas
- A multi-branch network that fuses mammogram images with clinical data, with an analysis of each branch's contribution.
Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.
Test your understanding
What is the benefit of a 1×1 convolution before the 5×5 in Inception?
Why? A bottleneck.
Four branches with 16, 32, 8 and 8 channels: how many output channels after concatenation?
Why? The sum.
What are the two branches in a two-stream video network?
Why? Simonyan and Zisserman 2014.
References
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1–9. https://doi.org/10.1109/CVPR.2015.7298594
Simonyan, K., & Zisserman, A. (2014). Two-stream convolutional networks for action recognition in videos. Advances in Neural Information Processing Systems (NeurIPS) 27. https://arxiv.org/abs/1406.2199
Xie, S., Girshick, R., Dollár, P., Tu, Z., & He, K. (2017). Aggregated residual transformations for deep neural networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 5987–5995. https://doi.org/10.1109/CVPR.2017.634
