Transformer Architectures

They rely on self-attention, where each element of the sequence computes a weight for every other element. The idea moved from machine translation to images (ViT, Swin and DeiT), to huge multimodal inputs (Perceiver), and to linking text with images (multimodal transformers and vision-language models). The cost of full attention grows with the square of the sequence length, which is what local windows and compressed latents address.

Advanced2020

Graph Transformer

Graph Transformer (GT)

Applying self-attention to the nodes of a graph with positional or structural encodings (such as Laplacian eigenvectors) so the model knows the graph's structure, going…

Graph Neural NetworksSpecialized ArchitecturesTransformer Architectures

Intermediate2021

Hybrid CNN–Transformer Networks

CNN-Transformer Hybrids

Designs that combine convolution (local bias, efficiency, generalisation with less data) with attention (global context, large capacity): usually a convolutional stem or…

Hybrid ArchitecturesTransformer Architectures

Intermediate2021

U-Net with Transformers

U-Net + Transformer (U-Net+Transformer)

A family of segmentation models that keeps the U-Net shape (an encoder–decoder with skip connections) and introduces the transformer somewhere: the bottleneck, the whole…

Hybrid ArchitecturesTransformer Architectures

Intermediate2021

TransUNet

TransUNet

A medical segmentation model that uses a hybrid encoder: a CNN (ResNet-50) extracts a feature map, which is split into patches and passed through a ViT, followed by a ca…

Hybrid ArchitecturesTransformer Architectures

Advanced2022

UNETR

UNETR

A 3D segmentation model whose encoder is a pure ViT on 16³ volumetric patches; representations are taken from its layers (3, 6, 9, 12), reshaped and upsampled with trans…

Hybrid ArchitecturesTransformer Architectures

Advanced2022

Swin UNETR

Swin UNETR (Swin-UNETR)

A volumetric segmentation model whose encoder is a hierarchical Swin Transformer (shifted local windows and patch merging) that gives features at five scales, connected…

Hybrid ArchitecturesTransformer Architectures

Advanced2020

Retrieval-augmented neural architectures

Retrieval-Augmented Neural Architectures (RAG)

Models that combine a non-parametric memory (a document index) with a parametric model: a retriever finds passages relevant to the question, and a generator uses them to…

Specialized ArchitecturesTransformer Architectures

Basic2017

Transformer

Transformer

An architecture based entirely on multi-head self-attention instead of recurrence or convolution, so it processes all sequence positions in parallel and links any two po…

Transformer Architectures

Intermediate2021

Vision Transformer

Vision Transformer (ViT)

Applies the standard Transformer encoder directly to images after splitting them into fixed-size patches treated as tokens, with a special classification token and a lea…

Transformer Architectures

Advanced2021

Swin Transformer

Swin Transformer

A hierarchical vision transformer that computes attention within non-overlapping local windows and shifts the windows in successive layers to exchange information betwee…

Transformer Architectures

Intermediate2021

Data-efficient image transformers

Data-efficient Image Transformers (DeiT)

A method for training ViT on ImageNet-1k alone with strong data augmentation and regularization, plus an extra "distillation token" that learns from the predictions of a…

Transformer Architectures

Advanced2021

Perceiver

Perceiver

A general architecture that reads a huge input of any type through cross-attention into a small fixed-size latent array, then processes the latents with self-attention,…

Transformer Architectures

Basic2017

Cross-attention

Cross-Attention (Cross-Attn)

Attention in which the queries come from one sequence while the keys and values come from another sequence or modality, so each element of the first reads what it needs…

Transformer Architectures

Intermediate2019

Multimodal transformers

Multimodal Transformers (MM-Transformer)

Transformers that process more than one modality (image, text, audio...) either in a single sequence with a modality-type embedding (single-stream), or in two streams th…

Transformer Architectures

Advanced2021

Vision-language models

Vision-Language Models (VLM)

Models that connect a visual encoder to a language model: either through contrastive alignment in a shared space (CLIP), or by turning visual features into tokens that a…

Transformer Architectures