Advanced Neural Architectures

Architectures that change the computing unit itself, not just the order of layers: learnable activation functions on the edges in KAN, vectors instead of scalars in capsules, timed spikes in spiking networks, differential equations in liquid networks, networks that generate the weights of others, equilibrium points instead of a fixed depth, and a fixed random reservoir in reservoir computing. Many of them are still an active research topic, so they are not a general replacement for the common architectures.

Advanced2024

Kolmogorov–Arnold Networks

Kolmogorov–Arnold Networks (KAN)

Networks that place learnable activation functions (B-spline curves) on the edges instead of fixed weights, while the nodes only sum their inputs, inspired by the Kolmog…

Advanced Neural Architectures

Intermediate2017

Capsule Networks

Capsule Networks (CapsNet)

Networks that represent entities as capsules (vectors) whose length expresses the probability that the entity is present and whose direction expresses its properties (po…

Advanced Neural Architectures

Intermediate1997

Spiking Neural Networks

Spiking Neural Networks (SNN)

Networks whose neurons communicate with discrete binary spikes in time, like the brain. The membrane potential accumulates and fires a spike when it exceeds the threshol…

Advanced Neural Architectures

Advanced2021

Liquid Neural Networks

Liquid Neural Networks (LNN)

Continuous-time recurrent networks in which each neuron's time constant changes with the input (Liquid Time-Constant), so their dynamics adapt to the data with a small n…

Temporal and Sequential NetworksAdvanced Neural Architectures

Intermediate2017

Hypernetworks

Hypernetworks

A small network (the hypernetwork) generates the weights of another network (the target network), so the weights become a function of a conditioning input such as the ta…

Advanced Neural ArchitecturesSpecialized Architectures

Advanced2019

Deep Equilibrium Models

Deep Equilibrium Models (DEQ)

Instead of stacking L layers, the model looks for a fixed point of a single layer z^*=f_θ(z^*,x), as if it were an infinitely deep network with shared weights, and compu…

Advanced Neural Architectures

Basic2004

Reservoir Computing

Reservoir Computing (ESN / LSM)

A large recurrent network with fixed random weights (the reservoir) turns the input into rich states, and only the linear readout layer is trained, usually with closed-f…

Temporal and Sequential NetworksAdvanced Neural Architectures