Architecture Comparator

Choose two or more architectures and compare the idea, structure, data, applications, strengths and limitations, and the parameter count of the example that was actually run.

Loading the lab… If it does not appear, make sure JavaScript is enabled in your browser.

The comparison shows no performance number that was not computed or documented: parameters and timings come from actually running the teaching examples, and the rest is description from the encyclopedia pages.

Architectures related to this lab

Intermediate2014

Generative Adversarial Networks

Generative Adversarial Networks (GAN)

A generative framework of two competing networks: a generator that turns random noise into samples, and a discriminator that tries to tell real samples from generated on…

Generative Networks

Advanced2020

Diffusion Models

Diffusion Models (DDPM)

They gradually corrupt the data by adding Gaussian noise over T steps, then a network learns to reverse this process by predicting the added noise, so a new example is g…

Generative Networks

Intermediate2019

Physics-Informed Neural Networks

Physics-Informed Neural Networks (PINN)

A network that approximates the solution of a partial differential equation u_θ(x,t) and is trained with a loss that combines the equation's residual (computed by automa…

Physics-Informed Neural Networks

Intermediate1991

Mixture of experts

Mixture of Experts (MoE)

Several "expert" networks and a gating network that decides, for each input, the weight of each expert in the output. The experts learn to specialize in different region…

Specialized Architectures

Advanced2017

Sparse mixture of experts

Sparse Mixture of Experts (Sparse MoE)

An expert layer in which only a small number of experts (top-k) is activated per token, so the model's parameter count grows greatly without a similar increase in per-to…

Specialized Architectures

Advanced2017

Neural architecture search

Neural Architecture Search (NAS)

Automating architecture design: a search space that defines possible architectures, a search strategy (reinforcement learning, evolution, Bayesian, gradient) that propos…

Specialized Architectures

Advanced2020

Retrieval-augmented neural architectures

Retrieval-Augmented Neural Architectures (RAG)

Models that combine a non-parametric memory (a document index) with a parametric model: a retriever finds passages relevant to the question, and a generator uses them to…

Specialized ArchitecturesTransformer Architectures

Basic2017

Transformer

Transformer

An architecture based entirely on multi-head self-attention instead of recurrence or convolution, so it processes all sequence positions in parallel and links any two po…

Transformer Architectures

Intermediate2021

Vision Transformer

Vision Transformer (ViT)

Applies the standard Transformer encoder directly to images after splitting them into fixed-size patches treated as tokens, with a special classification token and a lea…

Transformer Architectures

Advanced2021

Swin Transformer

Swin Transformer

A hierarchical vision transformer that computes attention within non-overlapping local windows and shifts the windows in successive layers to exchange information betwee…

Transformer Architectures