AdvancedSpecialized Architectures

Modular neural networks (Modular NN)

Networks built from reusable modules (such as Find, And and Count) composed dynamically according to the structure of the task or question, so skills transfer to new compositions not seen in training.

Overview

NameModular neural networks
Year introduced2016
ResearchersJacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein (Neural Module Networks)
InstitutionUC Berkeley
Original paperNeural module networks DOI arXiv
Data typeImages, Text, Multiple modalities
DomainsComputer vision, Robotics, Natural language processing
Computational costDepends on the program; usually small.
Data sizeMedium, with varied programs.

Problem it solves

Large monolithic networks memorize patterns and do not generalize compositionally with ease (e.g. they know "red" and "triangle" but fail on "red triangle" if they have not seen it).

Background and evolution

What came before: Multilayer Perceptron, Attention Mechanism

  1. 2016

    Neural Module Networks for visual question answering by composing modules according to the parse of the question. [ref]

  2. 2023

    The "Modular Deep Learning" survey classifies modules, routing and aggregation. [ref]

Core idea

Each module performs a simple operation with its own weights. A "layout" (program) determines how the modules are composed for each example: Count(And(Find[red], Find[triangle])). Because the same Find[red] module is used in every program, it learns its meaning independently, so it can be combined with other modules in new programs.

Example: A robot learns "pick up", "blue" and "cube" separately, then carries out "pick up the blue cube" without special training.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
ModulesSmall specialized modules
Layout / programDetermines the composition
Router or parserSelects the layout
Shared trainingAcross multiple programs

Equations

1. Module composition

$$\hat y=\mathrm{Count}\big(\mathrm{Find}_{a}(s)\odot\mathrm{Find}_{b}(s)\big)$$

Symbols
$\mathrm{Find}_a$Attention on cells with attribute a
$\odot$Intersection (And)
Dimensions
Attention $\in[0,1]^{9}$
Numerical example
Red attention $(1,0,1)$ and triangle attention $(1,1,0)$ ⇒ intersection $(1,0,0)$ and a count of 1.
Relation to the architecture
The program determines the composition for each question.

2. Flexible routing (general)

$$y=\sum_{m}r_m(x)\,f_m(x)$$

Symbols
$r_m$Routing score
$f_m$Module m
Dimensions
Output
Numerical example
Hard routing: $r=(0,1,0)$ selects only the second module.
Relation to the architecture
MoE is a special case with learned routing.

How it works, step by step

  1. Define a module library.
  2. Determine the layout for each example (given or learned).
  3. Execute the modules in order.
  4. Train on varied programs.
  5. Test on new compositions.

Strengths and limitations

Strengths

  • Compositional generalization.
  • Interpretability (clear steps).
  • Reuse and update individual modules.

Limitations

  • Needs layouts or a parser.
  • Learning the layout itself is hard.
  • Slower than a monolithic network.

When to use it, and when not?

Good for

  • Compositional tasks with clear structure.
  • Systems that need step-by-step explanation.

Not suited for

  • Perceptual tasks with no compositional structure.

Applications

DomainUse
Computer visionVisual question answering.
Natural language processingUnderstanding compound instructions, and composable adapters.
RoboticsComposing motor skills.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

Python (PyTorch)
# Neural Module Network: Find/And/Count modules and compositional generalization
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
ATTR = ['red', 'green', 'blue', 'circle', 'square', 'triangle']
class Find(nn.Module):
    """Find[a] module: gives attention (0..1) on the cells carrying attribute a. One module per attribute, reused in any program."""
    def __init__(self, d=7):
        super().__init__(); self.net = nn.Sequential(nn.Linear(d, 16), nn.ReLU(), nn.Linear(16, 1))
    def forward(self, scene):                        # scene: B×9×7
        return torch.sigmoid(self.net(scene)).squeeze(-1)    # B×9

class NeuralModuleNet(nn.Module):
    def __init__(self, max_count=9):
        super().__init__()
        self.find = nn.ModuleDict({a: Find() for a in ATTR})
        self.count = nn.Sequential(nn.Linear(1, 32), nn.ReLU(), nn.Linear(32, max_count + 1))   # Count module
    def run(self, program, scene):
        """program: ('count', 'red') or ('count', 'and', 'red', 'circle') — a layout that determines how modules are composed."""
        att = self.find[program[-1]](scene)
        if program[1] == 'and':
            att = att * self.find[program[2]](scene)          # And module: intersection of the two attentions
        return self.count(att.sum(1, keepdim=True))

# 3) Prepare the data
# 3×3 scenes: each cell is empty or holds a coloured shape. The question is a program of modules, and the answer is a count.
COLORS, SHAPES = ATTR[:3], ATTR[3:]
def make_scenes(n):
    s = torch.zeros(n, 9, 7)
    occ = torch.rand(n, 9) < 0.7
    col, shp = torch.randint(0, 3, (n, 9)), torch.randint(0, 3, (n, 9))
    s[..., 0] = (~occ).float()
    s.scatter_(2, (1 + col)[..., None], occ[..., None].float())
    s.scatter_(2, (4 + shp)[..., None], occ[..., None].float())
    return s
def answer(program, scene):
    def has(a):
        i = ATTR.index(a); return scene[..., 1 + i]
    m = has(program[-1]) * (has(program[2]) if program[1] == 'and' else 1)
    return m.sum(1).long()
# "Colour and shape" compositions for training; we hold out two compositions for testing to measure compositional generalization
ALL_AND = [('count', 'and', c, s) for c in COLORS for s in SHAPES]
HELD_OUT = [('count', 'and', 'red', 'triangle'), ('count', 'and', 'blue', 'circle')]
TRAIN_PROGRAMS = [('count', a) for a in ATTR] + [p for p in ALL_AND if p not in HELD_OUT]
model = NeuralModuleNet()

# 4) The loss and the optimizer
opt = torch.optim.Adam(model.parameters(), lr=3e-3)

# 5) Training loop: at each step a random training program and a batch of scenes
for step in range(3000):
    prog = TRAIN_PROGRAMS[torch.randint(0, len(TRAIN_PROGRAMS), (1,)).item()]
    scene = make_scenes(64)
    loss = F.cross_entropy(model.run(prog, scene), answer(prog, scene))          # Forward Pass through the program layout
    opt.zero_grad(); loss.backward(); opt.step()
    if step % 750 == 0 or step == 2999:
        print(f"step {step:4d}  loss={loss.item():.4f}")

# 6) Evaluation: training programs, then compositions never seen
@torch.no_grad()
def acc(programs, n=500):
    r = []
    for p in programs:
        s = make_scenes(n); r.append((model.run(p, s).argmax(1) == answer(p, s)).float().mean().item())
    return sum(r) / len(r)
print(f"accuracy on training programs        = {acc(TRAIN_PROGRAMS):.3f}")
print(f"accuracy on held-out compositions     = {acc(HELD_OUT):.3f}")

# 7) Inference: how many red triangles in a new scene? (a composition it was not trained on)
with torch.no_grad():
    s = make_scenes(1); p = ('count', 'and', 'red', 'triangle')
    print("program:", p, "→ predicted:", model.run(p, s).argmax(1).item(), " true:", answer(p, s).item())

This code was run on 2026-11-20 in 5.9 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

accuracy on training programs        = 0.996
accuracy on held-out compositions     = 1.000

Trainable parameters in the example: model = 1,264.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Intermediate1991

Mixture of experts

Mixture of Experts (MoE)

Several "expert" networks and a gating network that decides, for each input, the weight of each expert in the output. The experts learn to specialize in different region…

Specialized Architectures

Advanced2017

Sparse mixture of experts

Sparse Mixture of Experts (Sparse MoE)

An expert layer in which only a small number of experts (top-k) is activated per token, so the model's parameter count grows greatly without a similar increase in per-to…

Specialized Architectures

Intermediate2017

Hypernetworks

Hypernetworks

A small network (the hypernetwork) generates the weights of another network (the target network), so the weights become a function of a conditioning input such as the ta…

Advanced Neural ArchitecturesSpecialized Architectures

Advanced2020

Retrieval-augmented neural architectures

Retrieval-Augmented Neural Architectures (RAG)

Models that combine a non-parametric memory (a document index) with a parametric model: a retriever finds passages relevant to the question, and a generator uses them to…

Specialized ArchitecturesTransformer Architectures

Research ideas

  • Modular networks for answering compound clinical questions on radiology images, with an explanation of the reasoning steps.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    What is the main advantage of modular networks in the example?

  2. Concept

    What does the "layout" determine in an NMN?

  3. Concept

    How is the And module applied to two attentions?

References

  1. Andreas, J., Rohrbach, M., Darrell, T., & Klein, D. (2016). Neural module networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 39–48. https://doi.org/10.1109/CVPR.2016.12

    DOIarXivAutomatically verified via Crossref – 2026-11-20

  2. Pfeiffer, J., Ruder, S., Vulić, I., & Ponti, E. M. (2023). Modular deep learning. Transactions on Machine Learning Research (TMLR). https://arxiv.org/abs/2302.11529

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  3. Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991). Adaptive mixtures of local experts. Neural Computation, 3(1), 79–87. https://doi.org/10.1162/neco.1991.3.1.79

    DOIAutomatically verified via Crossref – 2026-11-20