Hypernetworks
Hypernetworks
A small network (the hypernetwork) generates the weights of another network (the target network), so the weights become a function of a conditioning input such as the ta…
Architectures that address a specific problem: routing each input to specialised experts to grow capacity without growing computation (MoE), composing reusable modules (modular networks), automated architecture search (NAS and DARTS), respecting the symmetries of the data (equivariant networks), modelling energy instead of probability directly, or connecting the model to an external knowledge store (RAG).
Hypernetworks
A small network (the hypernetwork) generates the weights of another network (the target network), so the weights become a function of a conditioning input such as the ta…
Graph Transformer (GT)
Applying self-attention to the nodes of a graph with positional or structural encodings (such as Laplacian eigenvectors) so the model knows the graph's structure, going…
Mixture of Experts (MoE)
Several "expert" networks and a gating network that decides, for each input, the weight of each expert in the output. The experts learn to specialize in different region…
Sparse Mixture of Experts (Sparse MoE)
An expert layer in which only a small number of experts (top-k) is activated per token, so the model's parameter count grows greatly without a similar increase in per-to…
Modular Neural Networks (Modular NN)
Networks built from reusable modules (such as Find, And and Count) composed dynamically according to the structure of the task or question, so skills transfer to new com…
Neural Architecture Search (NAS)
Automating architecture design: a search space that defines possible architectures, a search strategy (reinforcement learning, evolution, Bayesian, gradient) that propos…
Differentiable Architecture Search (DARTS)
Turns architecture search into a continuous optimization problem: each edge in the cell computes a weighted mixture of all possible operations with Softmax weights over…
Equivariant Neural Networks (ENN)
Networks built so that their output transforms in a specific way when the input is transformed by a symmetry (rotation, translation, reflection, permutation), so they do…
Energy-Based Models (EBM)
Models that learn a scalar energy function E_θ(x) that is low for likely configurations and high for others, and define the density implicitly p(x) e^-E(x) without compu…
Retrieval-Augmented Neural Architectures (RAG)
Models that combine a non-parametric memory (a document index) with a parametric model: a retriever finds passages relevant to the question, and a generator uses them to…
No results match the filters.
Sections
No section has this name. Press “Search” to search all site content.
Favourites
My results & certificates
My academy progressMy quiz resultsKids’ progressAcademy certificatesQuiz certificatesStart any lesson or quiz and it will show up here so you can get back to it quickly.
Settings
I am the smart assistant for deep learning architectures. Ask me about an architecture, an equation or code, or ask for a comparison, a path or an explanation of a lab result.
Answers from the encyclopedia's content only, with no language model and without sending your question to any outside party.
The assistant can make mistakes, so check the attached sources.