AdvancedSpecialized ArchitecturesTransformer Architectures

Retrieval-augmented neural architectures (RAG)

Models that combine a non-parametric memory (a document index) with a parametric model: a retriever finds passages relevant to the question, and a generator uses them to produce the answer, so knowledge can be updated by updating the index without retraining, and sources can be traced.

Overview

NameRetrieval-augmented neural architectures
Year introduced2020
ResearchersPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, et al.
InstitutionFacebook AI Research / University College London / New York University
Original paperRetrieval-augmented generation for knowledge-intensive NLP tasks arXiv
Data typeText, Multiple modalities
DomainsComputer vision, Medical image analysis, Natural language processing
Computational costApproximate search in the index + k generator passes.
Data sizeA large index; training on question–answer pairs.

Problem it solves

Language models store knowledge in their weights: it is hard to update, they hallucinate, and they do not cite their sources.

Background and evolution

What came before: Transformer, Attention Mechanism

  1. 2020

    REALM: pre-training with a latent retriever. [ref]

  2. 2020

    DPR: a dense retriever with two encoders. [ref]

  3. 2020

    RAG: a DPR retriever + a BART generator with marginalization over passages. [ref]

  4. 2022

    RETRO: retrieval from a database of trillions of tokens through cross-attention over chunks. [ref]

Core idea

$p(z\mid x)\propto\exp(d(z)^\top q(x))$ selects the top k passages. The generator computes $p(y\mid x,z)$ for each passage, and the result is marginalized over the passages. Because supervision is on the answer only, the retriever implicitly learns which passages help. Variants: RAG-Sequence (one passage for the whole answer), RAG-Token (a passage per token), and RETRO (retrieval inside the model layers).

Example: A medical assistant that answers a drug-dose question from the updated package leaflet, with a link to the source, instead of relying on the model's memory.

Interactive diagram

A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.

Select a layer in the diagram.

Components

ComponentRole
Document indexPassage vectors
Query encoder
Retriever (MIPS)Maximum inner product search
Generator / reader
Marginalization over passages

Equations

1. Dense retrieval

$$p_\eta(z\mid x)\propto\exp\big(d(z)^{\top}q(x)\big)$$

Symbols
$d(z)$Passage representation
$q(x)$Question representation
Dimensions
$d,q\in\mathbb{R}^{768}$ in DPR
Numerical example
Scores $(5,3,1)$ for three passages ⇒ $p\approx(0.867,0.117,0.016)$.
Relation to the architecture
Which passages are closest to the question.

2. RAG-Sequence

$$p(y\mid x)\approx\sum_{z\in\text{top-}k}p_\eta(z\mid x)\prod_{i}p_\theta(y_i\mid x,z,y_{<i})$$

Symbols
$p_\theta$The generator
Dimensions
Probability
Numerical example
$p(z)=(0.8,0.2)$ and $p(y\mid z)=(0.9,0.1)$ ⇒ $0.72+0.02=0.74$.
Relation to the architecture
Joint training without passage labels.

How it works, step by step

  1. Split the documents and index them.
  2. Encode the question.
  3. Retrieve the top k.
  4. Generate the answer conditioned on each passage.
  5. Marginalize or select.
  6. Show the sources.

Strengths and limitations

Strengths

  • Knowledge updates without retraining.
  • Ability to cite sources.
  • Less hallucination on knowledge questions, according to the authors' experiments.

Limitations

  • Quality is limited by retrieval quality.
  • The generator may ignore the passage and rely on its memory.
  • Latency and indexing cost.

When to use it, and when not?

Good for

  • Knowledge questions whose sources change.
  • Systems that need source tracing.

Not suited for

  • Tasks that do not depend on external knowledge.
  • An unreliable index.

Applications

DomainUse
Natural language processingOpen-domain question answering and assistants.
Medical image analysisAssistants based on up-to-date clinical evidence.
Computer visionRetrieval-augmented image generation.

PyTorch code

A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.

A small synthetic knowledge base; this encyclopedia's own assistant works on a similar retrieval principle over the encyclopedia's content.

Python (PyTorch)
# Mini RAG: bi-encoder retriever + reader + marginalization, compared with a closed-book model when knowledge is updated
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

# 2) Define the model
class Retriever(nn.Module):
    """A retriever using the dot product between the question and passage representations (like DPR). For simplicity the two encoders share the embedding table,
    so an entity's representation matches in the question and the passage even for entities never asked about in training (the original DPR uses separate BERT encoders)."""
    def __init__(self, vocab, d=32):
        super().__init__()
        self.q_enc = self.p_enc = nn.EmbeddingBag(vocab, d)
    def forward(self, q, passages):                       # q: B×Lq , passages: K×Lp
        return self.q_enc(q) @ self.p_enc(passages).T     # B×K relevance scores

class Reader(nn.Module):
    """The reader: generates the answer from the retrieved passage (here: classification over the value vocabulary). We made it read the passage alone;
    if it also read the question it would learn to "memorize" answers in its weights and ignore the passage, a known phenomenon that weakens grounding in the source."""
    def __init__(self, vocab, n_answers, d=32):
        super().__init__()
        self.emb = nn.EmbeddingBag(vocab, d)
        self.net = nn.Sequential(nn.Linear(d, 64), nn.ReLU(), nn.Linear(64, n_answers))
    def forward(self, q, passage):
        return self.net(self.emb(passage))

class RAG(nn.Module):
    """p(y|q) = Σ_{z ∈ top-k} p(z|q) · p(y|q, z)  — the RAG-Sequence form with marginalization over the retrieved passages."""
    def __init__(self, vocab, n_answers, k=4):
        super().__init__()
        self.retriever, self.reader, self.k = Retriever(vocab), Reader(vocab, n_answers), k
    def forward(self, q, kb):
        scores = self.retriever(q, kb)                                  # B×K
        top, idx = scores.topk(self.k, -1)
        p_z = top.softmax(-1)                                           # B×k
        logp_y = torch.stack([self.reader(q, kb[idx[:, j]]).log_softmax(-1) for j in range(self.k)], 1)   # B×k×A
        return torch.logsumexp(p_z.log()[..., None] + logp_y, 1), idx[:, 0]

# 3) Prepare the data
# A synthetic knowledge base: 60 entities, each with a "city" (one of 10). Passage = [entity, "located_in", city].
# Question = [random question template, entity]. Vocabulary: 0..59 entities, 60..69 cities, 70 "located_in", 71..74 question templates.
N_ENT, N_CITY = 60, 10
def build_kb(cities):
    return torch.stack([torch.arange(N_ENT), torch.full((N_ENT,), 70), 60 + cities], 1)   # K×3
cities = torch.randint(0, N_CITY, (N_ENT,))
KB = build_kb(cities)
def questions(ents):
    return torch.stack([71 + torch.randint(0, 4, (len(ents),)), ents], 1)
train_ents = torch.randint(0, 45, (4000,))           # training questions about the first 45 entities only
test_ents = torch.arange(45, 60)                     # entities never asked about during training
model = RAG(vocab=75, n_answers=N_CITY)
closed_book = nn.Sequential(nn.EmbeddingBag(75, 32), nn.Linear(32, 64), nn.ReLU(), nn.Linear(64, N_CITY))   # memorizes facts in its weights

# 4) Loss: −log p(y|q) after marginalizing over passages; the closed-book model trains on the same questions
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
opt_cb = torch.optim.Adam(closed_book.parameters(), lr=3e-3)

# 5) Training loop (supervision on the answer only; the retriever implicitly learns which passage helps)
for epoch in range(15):
    perm = torch.randperm(len(train_ents))
    for i in range(0, len(perm), 100):
        e = train_ents[perm[i:i + 100]]; q, y = questions(e), cities[e]
        logp, _ = model(q, KB)                                                  # Forward Pass: retrieval + reading
        loss = F.nll_loss(logp, y)
        opt.zero_grad(); loss.backward(); opt.step()
        loss_cb = F.cross_entropy(closed_book(q), y); opt_cb.zero_grad(); loss_cb.backward(); opt_cb.step()
    if epoch % 5 == 0 or epoch == 14:
        print(f"epoch {epoch:2d}  RAG loss={loss.item():.4f}  closed-book loss={loss_cb.item():.4f}")

# 6) Evaluation: (a) entities never asked about, (b) updating the knowledge base without retraining
@torch.no_grad()
def evaluate(kb, truth, ents):
    q = questions(ents)
    logp, top1 = model(q, kb)
    return ((logp.argmax(1) == truth[ents]).float().mean().item(), (top1 == ents).float().mean().item(),
            (closed_book(q).argmax(1) == truth[ents]).float().mean().item())
a, r, c = evaluate(KB, cities, torch.arange(45))
print(f"seen entities   : RAG acc={a:.3f} (retrieval@1={r:.3f}) | closed-book acc={c:.3f}")
a, r, c = evaluate(KB, cities, test_ents)
print(f"unseen entities : RAG acc={a:.3f} (retrieval@1={r:.3f}) | closed-book acc={c:.3f}")
new_cities = (cities + 1 + torch.randint(0, N_CITY - 1, (N_ENT,))) % N_CITY            # every fact changes
a, r, c = evaluate(build_kb(new_cities), new_cities, torch.arange(N_ENT))
print(f"after KB update : RAG acc={a:.3f} | closed-book acc={c:.3f}  (no retraining)")

# 7) Inference: a question about entity 10 with the retrieved passage
with torch.no_grad():
    q = questions(torch.tensor([10])); logp, top1 = model(q, KB)
    print("question tokens:", q[0].tolist(), "→ retrieved passage:", KB[top1[0]].tolist(), "→ answer city id:", logp.argmax(1).item(), "true:", cities[10].item())

This code was run on 2026-11-20 in 4.5 seconds (PyTorch (CPU), Python 3 – local build machine).

Sample output:

seen entities   : RAG acc=1.000 (retrieval@1=1.000) | closed-book acc=1.000
unseen entities : RAG acc=0.600 (retrieval@1=0.667) | closed-book acc=0.267
after KB update : RAG acc=0.867 | closed-book acc=0.017  (no retraining)

Trainable parameters in the example: model = 7,562, closed_book = 5,162.

These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.

Try it and go deeper

Related architectures

Basic2017

Transformer

Transformer

An architecture based entirely on multi-head self-attention instead of recurrence or convolution, so it processes all sequence positions in parallel and links any two po…

Transformer Architectures

Basic2017

Cross-attention

Cross-Attention (Cross-Attn)

Attention in which the queries come from one sequence while the keys and values come from another sequence or modality, so each element of the first reads what it needs…

Transformer Architectures

Advanced2021

Vision-language models

Vision-Language Models (VLM)

Models that connect a visual encoder to a language model: either through contrastive alignment in a shared space (CLIP), or by turning visual features into tokens that a…

Transformer Architectures

Advanced2016

Modular neural networks

Modular Neural Networks (Modular NN)

Networks built from reusable modules (such as Find, And and Count) composed dynamically according to the structure of the task or question, so skills transfer to new com…

Specialized Architectures

Research ideas

  • Arabic RAG for medical guidelines, with an evaluation of source grounding and hallucination rate.

Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.

Test your understanding

  1. Comparison

    What is the main advantage of RAG over a "closed-book" model?

  2. Mathematical

    With $p(z)=(0.6,0.4)$ and $p(y\mid z)=(0.5,1.0)$, what is $p(y)$?

  3. Concept

    What problem may appear if the generator reads the question and learns to memorize the answers?

References

  1. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems (NeurIPS) 33. https://arxiv.org/abs/2005.11401

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  2. Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020). REALM: Retrieval-augmented language model pre-training. International Conference on Machine Learning (ICML), PMLR 119, 3929–3938. https://arxiv.org/abs/2002.08909

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20

  3. Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-t. (2020). Dense passage retrieval for open-domain question answering. Proceedings of EMNLP 2020, 6769–6781. https://doi.org/10.18653/v1/2020.emnlp-main.550

    DOIarXivAutomatically verified via Crossref – 2026-11-20

  4. Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., et al. (2022). Improving language models by retrieving from trillions of tokens. International Conference on Machine Learning (ICML), PMLR 162, 2206–2240. https://arxiv.org/abs/2112.04426

    arXivAutomatically verified via OpenAlex/arXiv – 2026-11-20