Retrieval-augmented neural architectures (RAG)
Models that combine a non-parametric memory (a document index) with a parametric model: a retriever finds passages relevant to the question, and a generator uses them to produce the answer, so knowledge can be updated by updating the index without retraining, and sources can be traced.
Overview
| Name | Retrieval-augmented neural architectures |
|---|---|
| Year introduced | 2020 |
| Researchers | Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, et al. |
| Institution | Facebook AI Research / University College London / New York University |
| Original paper | Retrieval-augmented generation for knowledge-intensive NLP tasks arXiv |
| Data type | Text, Multiple modalities |
| Domains | Computer vision, Medical image analysis, Natural language processing |
| Computational cost | Approximate search in the index + k generator passes. |
| Data size | A large index; training on question–answer pairs. |
Problem it solves
Language models store knowledge in their weights: it is hard to update, they hallucinate, and they do not cite their sources.
Background and evolution
What came before: Transformer, Attention Mechanism
Core idea
$p(z\mid x)\propto\exp(d(z)^\top q(x))$ selects the top k passages. The generator computes $p(y\mid x,z)$ for each passage, and the result is marginalized over the passages. Because supervision is on the answer only, the retriever implicitly learns which passages help. Variants: RAG-Sequence (one passage for the whole answer), RAG-Token (a passage per token), and RETRO (retrieval inside the model layers).
Interactive diagram
A simplified illustrative diagram with the dimensions of a small example, not a drawing matching every detail of the original implementation. Click any layer to see its role, dimensions and mathematical operation.
Select a layer in the diagram.
Components
| Component | Role |
|---|---|
| Document index | Passage vectors |
| Query encoder | |
| Retriever (MIPS) | Maximum inner product search |
| Generator / reader | |
| Marginalization over passages |
Equations
1. Dense retrieval
| $d(z)$ | Passage representation |
| $q(x)$ | Question representation |
2. RAG-Sequence
| $p_\theta$ | The generator |
How it works, step by step
- Split the documents and index them.
- Encode the question.
- Retrieve the top k.
- Generate the answer conditioned on each passage.
- Marginalize or select.
- Show the sources.
Strengths and limitations
Strengths
- Knowledge updates without retraining.
- Ability to cite sources.
- Less hallucination on knowledge questions, according to the authors' experiments.
Limitations
- Quality is limited by retrieval quality.
- The generator may ignore the passage and rely on its memory.
- Latency and indexing cost.
When to use it, and when not?
Good for
- Knowledge questions whose sources change.
- Systems that need source tracing.
Not suited for
- Tasks that do not depend on external knowledge.
- An unreliable index.
Applications
| Domain | Use |
|---|---|
| Natural language processing | Open-domain question answering and assistants. |
| Medical image analysis | Assistants based on up-to-date clinical evidence. |
| Computer vision | Retrieval-augmented image generation. |
PyTorch code
A simplified teaching code sample that runs on small synthetic data, covering: imports, model definition, data, forward pass, loss, optimizer, training loop, evaluation and inference. It is not the paper's full implementation.
A small synthetic knowledge base; this encyclopedia's own assistant works on a similar retrieval principle over the encyclopedia's content.
# Mini RAG: bi-encoder retriever + reader + marginalization, compared with a closed-book model when knowledge is updated
# A simplified educational example in PyTorch (not the paper's full original implementation): small synthetic data,
# runs on the CPU in seconds. The numbers it prints belong to this example only.
# 1) Import the libraries
import torch
import torch.nn as nn
import torch.nn.functional as F
torch.manual_seed(0)
# 2) Define the model
class Retriever(nn.Module):
"""A retriever using the dot product between the question and passage representations (like DPR). For simplicity the two encoders share the embedding table,
so an entity's representation matches in the question and the passage even for entities never asked about in training (the original DPR uses separate BERT encoders)."""
def __init__(self, vocab, d=32):
super().__init__()
self.q_enc = self.p_enc = nn.EmbeddingBag(vocab, d)
def forward(self, q, passages): # q: B×Lq , passages: K×Lp
return self.q_enc(q) @ self.p_enc(passages).T # B×K relevance scores
class Reader(nn.Module):
"""The reader: generates the answer from the retrieved passage (here: classification over the value vocabulary). We made it read the passage alone;
if it also read the question it would learn to "memorize" answers in its weights and ignore the passage, a known phenomenon that weakens grounding in the source."""
def __init__(self, vocab, n_answers, d=32):
super().__init__()
self.emb = nn.EmbeddingBag(vocab, d)
self.net = nn.Sequential(nn.Linear(d, 64), nn.ReLU(), nn.Linear(64, n_answers))
def forward(self, q, passage):
return self.net(self.emb(passage))
class RAG(nn.Module):
"""p(y|q) = Σ_{z ∈ top-k} p(z|q) · p(y|q, z) — the RAG-Sequence form with marginalization over the retrieved passages."""
def __init__(self, vocab, n_answers, k=4):
super().__init__()
self.retriever, self.reader, self.k = Retriever(vocab), Reader(vocab, n_answers), k
def forward(self, q, kb):
scores = self.retriever(q, kb) # B×K
top, idx = scores.topk(self.k, -1)
p_z = top.softmax(-1) # B×k
logp_y = torch.stack([self.reader(q, kb[idx[:, j]]).log_softmax(-1) for j in range(self.k)], 1) # B×k×A
return torch.logsumexp(p_z.log()[..., None] + logp_y, 1), idx[:, 0]
# 3) Prepare the data
# A synthetic knowledge base: 60 entities, each with a "city" (one of 10). Passage = [entity, "located_in", city].
# Question = [random question template, entity]. Vocabulary: 0..59 entities, 60..69 cities, 70 "located_in", 71..74 question templates.
N_ENT, N_CITY = 60, 10
def build_kb(cities):
return torch.stack([torch.arange(N_ENT), torch.full((N_ENT,), 70), 60 + cities], 1) # K×3
cities = torch.randint(0, N_CITY, (N_ENT,))
KB = build_kb(cities)
def questions(ents):
return torch.stack([71 + torch.randint(0, 4, (len(ents),)), ents], 1)
train_ents = torch.randint(0, 45, (4000,)) # training questions about the first 45 entities only
test_ents = torch.arange(45, 60) # entities never asked about during training
model = RAG(vocab=75, n_answers=N_CITY)
closed_book = nn.Sequential(nn.EmbeddingBag(75, 32), nn.Linear(32, 64), nn.ReLU(), nn.Linear(64, N_CITY)) # memorizes facts in its weights
# 4) Loss: −log p(y|q) after marginalizing over passages; the closed-book model trains on the same questions
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
opt_cb = torch.optim.Adam(closed_book.parameters(), lr=3e-3)
# 5) Training loop (supervision on the answer only; the retriever implicitly learns which passage helps)
for epoch in range(15):
perm = torch.randperm(len(train_ents))
for i in range(0, len(perm), 100):
e = train_ents[perm[i:i + 100]]; q, y = questions(e), cities[e]
logp, _ = model(q, KB) # Forward Pass: retrieval + reading
loss = F.nll_loss(logp, y)
opt.zero_grad(); loss.backward(); opt.step()
loss_cb = F.cross_entropy(closed_book(q), y); opt_cb.zero_grad(); loss_cb.backward(); opt_cb.step()
if epoch % 5 == 0 or epoch == 14:
print(f"epoch {epoch:2d} RAG loss={loss.item():.4f} closed-book loss={loss_cb.item():.4f}")
# 6) Evaluation: (a) entities never asked about, (b) updating the knowledge base without retraining
@torch.no_grad()
def evaluate(kb, truth, ents):
q = questions(ents)
logp, top1 = model(q, kb)
return ((logp.argmax(1) == truth[ents]).float().mean().item(), (top1 == ents).float().mean().item(),
(closed_book(q).argmax(1) == truth[ents]).float().mean().item())
a, r, c = evaluate(KB, cities, torch.arange(45))
print(f"seen entities : RAG acc={a:.3f} (retrieval@1={r:.3f}) | closed-book acc={c:.3f}")
a, r, c = evaluate(KB, cities, test_ents)
print(f"unseen entities : RAG acc={a:.3f} (retrieval@1={r:.3f}) | closed-book acc={c:.3f}")
new_cities = (cities + 1 + torch.randint(0, N_CITY - 1, (N_ENT,))) % N_CITY # every fact changes
a, r, c = evaluate(build_kb(new_cities), new_cities, torch.arange(N_ENT))
print(f"after KB update : RAG acc={a:.3f} | closed-book acc={c:.3f} (no retraining)")
# 7) Inference: a question about entity 10 with the retrieved passage
with torch.no_grad():
q = questions(torch.tensor([10])); logp, top1 = model(q, KB)
print("question tokens:", q[0].tolist(), "→ retrieved passage:", KB[top1[0]].tolist(), "→ answer city id:", logp.argmax(1).item(), "true:", cities[10].item())
This code was run on 2026-11-20 in 4.5 seconds (PyTorch (CPU), Python 3 – local build machine).
Sample output:
seen entities : RAG acc=1.000 (retrieval@1=1.000) | closed-book acc=1.000 unseen entities : RAG acc=0.600 (retrieval@1=0.667) | closed-book acc=0.267 after KB update : RAG acc=0.867 | closed-book acc=0.017 (no retraining)
Trainable parameters in the example: model = 7,562, closed_book = 5,162.
These results are for small synthetic data and few epochs, and only prove that the code runs; do not compare them with published results.
Try it and go deeper
Related architectures
Transformer
Transformer
An architecture based entirely on multi-head self-attention instead of recurrence or convolution, so it processes all sequence positions in parallel and links any two po…
Cross-attention
Cross-Attention (Cross-Attn)
Attention in which the queries come from one sequence while the keys and values come from another sequence or modality, so each element of the first reads what it needs…
Vision-language models
Vision-Language Models (VLM)
Models that connect a visual encoder to a language model: either through contrastive alignment in a shared space (CLIP), or by turning visual features into tokens that a…
Modular neural networks
Modular Neural Networks (Modular NN)
Networks built from reusable modules (such as Find, And and Count) composed dynamically according to the structure of the task or question, so skills transfer to new com…
No results match the filters.
Research ideas
- Arabic RAG for medical guidelines, with an evaluation of source grounding and hallucination rate.
Starting suggestions, not confirmed research gaps; review the recent literature before adopting them.
Test your understanding
What is the main advantage of RAG over a "closed-book" model?
Why? The memory is non-parametric.
With $p(z)=(0.6,0.4)$ and $p(y\mid z)=(0.5,1.0)$, what is $p(y)$?
Why? $0.3+0.4$.
What problem may appear if the generator reads the question and learns to memorize the answers?
Why? A known phenomenon in RAG systems.
References
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems (NeurIPS) 33. https://arxiv.org/abs/2005.11401
Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020). REALM: Retrieval-augmented language model pre-training. International Conference on Machine Learning (ICML), PMLR 119, 3929–3938. https://arxiv.org/abs/2002.08909
Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-t. (2020). Dense passage retrieval for open-domain question answering. Proceedings of EMNLP 2020, 6769–6781. https://doi.org/10.18653/v1/2020.emnlp-main.550
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., et al. (2022). Improving language models by retrieving from trillions of tokens. International Conference on Machine Learning (ICML), PMLR 162, 2206–2240. https://arxiv.org/abs/2112.04426
