Hypernetworks (HyperNet)
A small network (the hypernetwork) generates the weights of another network (the target network), so the weights become a function of a conditioning input such as the task identity or the layer number.
Overview
| Name | Hypernetworks |
|---|---|
| Year introduced | 2017 |
| Researchers | David Ha, Andrew Dai, Quoc V. Le |
| Institution | Google Brain |
| Original paper | HyperNetworks arXiv |
| Data type | Multiple modalities |
| Domains | Automated machine learning (AutoML), Computer vision, Natural language processing |
Problem it solves
Storing independent weights for each layer or each task is costly, and sharing all weights restricts flexibility. We need a middle way that generates different weights from a compact representation.
Background and evolution
What came before: Multilayer Perceptron, Recurrent neural network
- 2017
Ha, Dai and Le introduced static hypernetworks (for CNNs) and dynamic ones (for LSTMs) at ICLR 2017. [ref]
Core idea
Instead of learning $W$ directly, we learn a function $W=h_\phi(z)$ where $z$ is a small embedding. For convolutional networks each layer has an embedding $z_j$ from which the hypernetwork generates its kernel. In the dynamic form $z$ changes at each time step, so the LSTM weights change over time.
Interactive diagram
A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.
Select a layer in the diagram.
Components
| Component | Role |
|---|---|
| Conditioning input z | An embedding of a task, layer or time step |
| Hypernetwork h_φ | An MLP that generates the weight vector |
| Target network | A network that uses the generated weights |
| Training | The gradient passes through the generated weights to φ |
Equations
1. Weight generation
| $h_\phi$ | The hypernetwork |
| $z$ | The conditioning embedding |
| $f$ | The target network |
2. Gradient through the generated weights
| $\partial\mathcal{L}/\partial\theta$ | Gradient of the loss with respect to the target's weights |
How it works, step by step
- The conditioning embedding z is chosen.
- The hypernetwork generates the weight vector.
- It is reshaped into the target's matrices.
- The target network is run on the input.
- The hypernetwork's parameters and the embeddings are updated.
Strengths and limitations
Strengths
- Sharing knowledge between tasks or layers.
- Parameter compression.
- A basis for multi-task learning, meta-learning and continual learning.
Limitations
- Training can be unstable.
- Generating the weights of large networks needs tricks (splitting into chunks).
When to use it, and when not?
Good for
- Multi-task learning and continual learning.
- Adapting a model to an external condition.
Not suited for
- A single simple task, where generation brings no benefit.
Applications
| Domain | Use |
|---|---|
| Automated machine learning (AutoML) | Meta-learning and fast adaptation to new tasks. |
| Natural language processing | Time-varying weights for LSTM (HyperLSTM). |
| Computer vision | Generating convolution kernels and compressing networks. |
PyTorch code
A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.
# A hypernetwork that generates the weights of an MLP for each task
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
# 2) Define the model
class HyperMLP(nn.Module):
"""Target: an MLP 1→16→1. The hypernetwork generates all of its parameters (49) from the task embedding."""
def __init__(self, n_tasks, k=4, h=16):
super().__init__()
self.h = h
self.task_emb = nn.Embedding(n_tasks, k)
self.P = h + h + h + 1 # W1(1×h)+b1+W2(h×1)+b2
self.hyper = nn.Sequential(nn.Linear(k, 64), nn.ReLU(), nn.Linear(64, self.P))
def forward(self, x, task): # x: B×1, task: B
p = self.hyper(self.task_emb(task)) # B×P weights per sample
h = self.h
W1, b1, W2, b2 = p[:, :h], p[:, h:2 * h], p[:, 2 * h:3 * h], p[:, 3 * h:]
z = torch.tanh(x * W1 + b1) # the target network with the generated weights
return (z * W2).sum(1, keepdim=True) + b2
# 3) Prepare the data
# 6 tasks: y = a_t · sin(x + φ_t); each task has a different amplitude and phase
n_tasks = 6
amp, phase = torch.rand(n_tasks) * 1.5 + 0.5, torch.rand(n_tasks) * 3
task = torch.randint(0, n_tasks, (1500,))
x = torch.rand(1500, 1) * 6 - 3
y = amp[task, None] * torch.sin(x + phase[task, None])
X = torch.cat([x, task[:, None].float()], 1) # we pass the task number with the input
X_train, y_train, X_test, y_test = X[:1200], y[:1200], X[1200:], y[1200:]
model = HyperMLP(n_tasks)
# 4) Loss function and optimizer
loss_fn = nn.MSELoss()
opt = torch.optim.Adam(model.parameters(), lr=5e-3)
# 5) Training loop with the forward pass
EPOCHS, BATCH = 40, 64
for epoch in range(EPOCHS):
model.train()
perm = torch.randperm(len(X_train))
total = 0.0
for i in range(0, len(X_train), BATCH):
idx = perm[i:i + BATCH]
xb, yb = X_train[idx], y_train[idx]
out = model(xb[:, :1], xb[:, 1].long()) # Forward Pass
loss = loss_fn(out, yb)
opt.zero_grad()
loss.backward() # Backpropagation
opt.step()
total += loss.item() * len(idx)
if epoch % max(1, EPOCHS // 4) == 0 or epoch == EPOCHS - 1:
print(f"epoch {epoch:3d} loss={total / len(X_train):.4f}")
# 6) Evaluation
model.eval()
with torch.no_grad():
xb = X_test
out = model(xb[:, :1], xb[:, 1].long())
mse = F.mse_loss(out, y_test).item()
print(f"test MSE = {mse:.4f}")
# 7) Inference on a new sample
with torch.no_grad():
xb = X_test[:1]
out = model(xb[:, :1], xb[:, 1].long())
print("prediction:", out.squeeze().tolist())
This code was run on 2026-11-20 in 3.3 seconds (PyTorch (CPU), Python 3 – local build machine).
Sample output:
test MSE = 0.0010
Trainable parameters in the example: model = 3,529.
These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.
Try it and go deeper
Related architectures
Modular neural networks
Modular Neural Networks (Modular NN)
Networks built from reusable modules (such as Find, And and Count) composed dynamically according to the structure of the task or question, so skills transfer to new com…
Mixture of experts
Mixture of Experts (MoE)
Several "expert" networks and a gating network that decides, for each input, the weight of each expert in the output. The experts learn to specialize in different region…
Neural architecture search
Neural Architecture Search (NAS)
Automating architecture design: a search space that defines possible architectures, a search strategy (reinforcement learning, evolution, Bayesian, gradient) that propos…
No results match the filters.
Research ideas
- Use hypernetworks in continual learning to classify medical images from several hospitals while measuring forgetting.
Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.
Test your understanding
What does the hypernetwork generate?
Why? This is its basic definition.
How are the hypernetwork's parameters trained?
Why? The generated weights are a differentiable function of φ.
What is the benefit of generating different weights for each task from a small embedding?
Why? The small embedding selects a "version" of the weights.
References
Ha, D., Dai, A., & Le, Q. V. (2017). HyperNetworks. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1609.09106
