IntermediateAdvanced Neural ArchitecturesSpecialized Architectures

Hypernetworks (HyperNet)

A small network (the hypernetwork) generates the weights of another network (the target network), so the weights become a function of a conditioning input such as the task identity or the layer number.

Overview

NameHypernetworks
Year introduced2017
ResearchersDavid Ha, Andrew Dai, Quoc V. Le
InstitutionGoogle Brain
Original paperHyperNetworks arXiv
Data typeMultiple modalities
DomainsAutomated machine learning (AutoML), Computer vision, Natural language processing

Problem it solves

Storing independent weights for each layer or each task is costly, and sharing all weights restricts flexibility. We need a middle way that generates different weights from a compact representation.

Background and evolution

What came before: Multilayer Perceptron, Recurrent neural network

  1. 2017

    Ha, Dai and Le introduced static hypernetworks (for CNNs) and dynamic ones (for LSTMs) at ICLR 2017. [ref]

Core idea

Instead of learning $W$ directly, we learn a function $W=h_\phi(z)$ where $z$ is a small embedding. For convolutional networks each layer has an embedding $z_j$ from which the hypernetwork generates its kernel. In the dynamic form $z$ changes at each time step, so the LSTM weights change over time.

Example: You have 5 similar regression tasks; the model learns an embedding for each task, and the hypernetwork generates the weights of the suitable regressor for each task from its embedding.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
Conditioning input zAn embedding of a task, layer or time step
Hypernetwork h_φAn MLP that generates the weight vector
Target networkA network that uses the generated weights
TrainingThe gradient passes through the generated weights to φ

Equations

1. Weight generation

$$\theta_{\text{target}}=h_\phi(z),\qquad \hat y=f\!\left(x;\,h_\phi(z)\right)$$

Symbols
$h_\phi$The hypernetwork
$z$The conditioning embedding
$f$The target network
Dimensions
$z\in\mathbb{R}^{k}$, $\theta\in\mathbb{R}^{P}$ with $k\ll P$
Numerical example
$k=4$ and a target network with 65 parameters: a linear hypernetwork needs $4\times65+65=325$ parameters to generate it for any task.
Relation to the architecture
It separates "where the weights come from" from "how they are used".

2. Gradient through the generated weights

$$\frac{\partial\mathcal{L}}{\partial\phi}=\frac{\partial\mathcal{L}}{\partial\theta}\,\frac{\partial h_\phi(z)}{\partial\phi}$$

Symbols
$\partial\mathcal{L}/\partial\theta$Gradient of the loss with respect to the target's weights
Dimensions
Same shape as $\phi$
Numerical example
Computed automatically with autograd.
Relation to the architecture
It shows that the hypernetwork is trained end to end.

How it works, step by step

  1. The conditioning embedding z is chosen.
  2. The hypernetwork generates the weight vector.
  3. It is reshaped into the target's matrices.
  4. The target network is run on the input.
  5. The hypernetwork's parameters and the embeddings are updated.

Strengths and limitations

Strengths

  • Sharing knowledge between tasks or layers.
  • Parameter compression.
  • A basis for multi-task learning, meta-learning and continual learning.

Limitations

  • Training can be unstable.
  • Generating the weights of large networks needs tricks (splitting into chunks).

When to use it, and when not?

Good for

  • Multi-task learning and continual learning.
  • Adapting a model to an external condition.

Not suited for

  • A single simple task, where generation brings no benefit.

Applications

DomainUse
Automated machine learning (AutoML)Meta-learning and fast adaptation to new tasks.
Natural language processingTime-varying weights for LSTM (HyperLSTM).
Computer visionGenerating convolution kernels and compressing networks.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

Python (PyTorch)
# A hypernetwork that generates the weights of an MLP for each task
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
class HyperMLP(nn.Module):
    """Target: an MLP 1→16→1. The hypernetwork generates all of its parameters (49) from the task embedding."""
    def __init__(self, n_tasks, k=4, h=16):
        super().__init__()
        self.h = h
        self.task_emb = nn.Embedding(n_tasks, k)
        self.P = h + h + h + 1                                  # W1(1×h)+b1+W2(h×1)+b2
        self.hyper = nn.Sequential(nn.Linear(k, 64), nn.ReLU(), nn.Linear(64, self.P))
    def forward(self, x, task):                                 # x: B×1, task: B
        p = self.hyper(self.task_emb(task))                     # B×P weights per sample
        h = self.h
        W1, b1, W2, b2 = p[:, :h], p[:, h:2 * h], p[:, 2 * h:3 * h], p[:, 3 * h:]
        z = torch.tanh(x * W1 + b1)                             # the target network with the generated weights
        return (z * W2).sum(1, keepdim=True) + b2

# 3) Prepare the data
# 6 tasks: y = a_t · sin(x + φ_t); each task has a different amplitude and phase
n_tasks = 6
amp, phase = torch.rand(n_tasks) * 1.5 + 0.5, torch.rand(n_tasks) * 3
task = torch.randint(0, n_tasks, (1500,))
x = torch.rand(1500, 1) * 6 - 3
y = amp[task, None] * torch.sin(x + phase[task, None])
X = torch.cat([x, task[:, None].float()], 1)                    # we pass the task number with the input
X_train, y_train, X_test, y_test = X[:1200], y[:1200], X[1200:], y[1200:]
model = HyperMLP(n_tasks)

# 4) Loss function and optimizer
loss_fn = nn.MSELoss()
opt = torch.optim.Adam(model.parameters(), lr=5e-3)

# 5) Training loop with the forward pass
EPOCHS, BATCH = 40, 64
for epoch in range(EPOCHS):
    model.train()
    perm = torch.randperm(len(X_train))
    total = 0.0
    for i in range(0, len(X_train), BATCH):
        idx = perm[i:i + BATCH]
        xb, yb = X_train[idx], y_train[idx]
        out = model(xb[:, :1], xb[:, 1].long())                       # Forward Pass
        loss = loss_fn(out, yb)
        opt.zero_grad()
        loss.backward()                    # Backpropagation
        opt.step()
        total += loss.item() * len(idx)
    if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
        print(f"epoch {epoch:3d}  loss={total / len(X_train):.4f}")

# 6) Evaluation
model.eval()
with torch.no_grad():
    xb = X_test
    out = model(xb[:, :1], xb[:, 1].long())
    mse = F.mse_loss(out, y_test).item()
    print(f"test MSE = {mse:.4f}")

# 7) Inference on a new sample
with torch.no_grad():
    xb = X_test[:1]
    out = model(xb[:, :1], xb[:, 1].long())
    print("prediction:", out.squeeze().tolist())

This code was run on 2026-11-20 in 3.3 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

test MSE = 0.0010

Trainable parameters in the example: model = 3,529.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Advanced2016

Modular neural networks

Modular Neural Networks (Modular NN)

Networks built from reusable modules (such as Find, And and Count) composed dynamically according to the structure of the task or question, so skills transfer to new com…

Specialized Architectures

Intermediate1991

Mixture of experts

Mixture of Experts (MoE)

Several "expert" networks and a gating network that decides, for each input, the weight of each expert in the output. The experts learn to specialize in different region…

Specialized Architectures

Advanced2017

Neural architecture search

Neural Architecture Search (NAS)

Automating architecture design: a search space that defines possible architectures, a search strategy (reinforcement learning, evolution, Bayesian, gradient) that propos…

Specialized Architectures

Research ideas

  • Use hypernetworks in continual learning to classify medical images from several hospitals while measuring forgetting.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Concept

    What does the hypernetwork generate?

  2. Concept

    How are the hypernetwork's parameters trained?

  3. Concept

    What is the benefit of generating different weights for each task from a small embedding?

References

  1. Ha, D., Dai, A., & Le, Q. V. (2017). HyperNetworks. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1609.09106

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20