Specialized Architectures

Architectures that address a specific problem: routing each input to specialised experts to grow capacity without growing computation (MoE), composing reusable modules (modular networks), automated architecture search (NAS and DARTS), respecting the symmetries of the data (equivariant networks), modelling energy instead of probability directly, or connecting the model to an external knowledge store (RAG).

Intermediate2017

Hypernetworks

Hypernetworks

A small network (the hypernetwork) generates the weights of another network (the target network), so the weights become a function of a conditioning input such as the ta…

Advanced Neural ArchitecturesSpecialized Architectures

Advanced2020

Graph Transformer

Graph Transformer (GT)

Applying self-attention to the nodes of a graph with positional or structural encodings (such as Laplacian eigenvectors) so the model knows the graph's structure, going…

Graph Neural NetworksSpecialized ArchitecturesTransformer Architectures

Intermediate1991

Mixture of experts

Mixture of Experts (MoE)

Several "expert" networks and a gating network that decides, for each input, the weight of each expert in the output. The experts learn to specialize in different region…

Specialized Architectures

Advanced2017

Sparse mixture of experts

Sparse Mixture of Experts (Sparse MoE)

An expert layer in which only a small number of experts (top-k) is activated per token, so the model's parameter count grows greatly without a similar increase in per-to…

Specialized Architectures

Advanced2016

Modular neural networks

Modular Neural Networks (Modular NN)

Networks built from reusable modules (such as Find, And and Count) composed dynamically according to the structure of the task or question, so skills transfer to new com…

Specialized Architectures

Advanced2017

Neural architecture search

Neural Architecture Search (NAS)

Automating architecture design: a search space that defines possible architectures, a search strategy (reinforcement learning, evolution, Bayesian, gradient) that propos…

Specialized Architectures

Research2019

Differentiable architecture search

Differentiable Architecture Search (DARTS)

Turns architecture search into a continuous optimization problem: each edge in the cell computes a weighted mixture of all possible operations with Softmax weights over…

Specialized Architectures

Research2016

Equivariant neural networks

Equivariant Neural Networks (ENN)

Networks built so that their output transforms in a specific way when the input is transformed by a symmetry (rotation, translation, reflection, permutation), so they do…

Specialized Architectures

Advanced2006

Energy-based models

Energy-Based Models (EBM)

Models that learn a scalar energy function E_θ(x) that is low for likely configurations and high for others, and define the density implicitly p(x) e^-E(x) without compu…

Generative NetworksSpecialized Architectures

Advanced2020

Retrieval-augmented neural architectures

Retrieval-Augmented Neural Architectures (RAG)

Models that combine a non-parametric memory (a document index) with a parametric model: a retriever finds passages relevant to the question, and a generator uses them to…

Specialized ArchitecturesTransformer Architectures