Retrieval-Augmented Generation (RAG): How It Works and How to Evaluate It

Methods & evaluation · By the Parallel Text editorial team · Published 2026-11-02

The domed main reading room of the Library of Congress, with concentric rings of desks around a central circulation desk — The Main Reading Room of the Library of Congress, Washington. Photo: Carol M. Highsmith, public domain.

A language model answers from what it absorbed during training. That knowledge is frozen at a date, hard to inspect and impossible to cite. Retrieval-augmented generation, usually shortened to RAG, adds a step: before generating, the system searches a collection of documents and gives the most relevant passages to the model as context. The idea is simple. Getting it to work reliably, and proving that it does, is where most of the research and engineering effort goes.

Where the term comes from

The name was introduced by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University in "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (NeurIPS 2020). Their system paired a pretrained sequence-to-sequence model (BART) with a dense passage retriever over an index of Wikipedia, and trained the two jointly for question answering and fact verification. They described two variants: one that conditions the whole answer on the same set of retrieved passages, and one that can draw on different passages for each generated token.

The underlying idea is older. Open-domain question answering systems had long chained a search engine with a reading component, and the same year saw REALM (Guu and colleagues, 2020), which built retrieval into language model pretraining. What changed after 2022 is scale and packaging: with instruction-tuned chat models, retrieval no longer needs joint training. Most deployed "RAG" systems today retrieve with one model and generate with another, connected only by a prompt.

The pipeline, step by step

Ingestion. Documents are cleaned, converted to text (PDF extraction alone causes many errors), and split into chunks.
Indexing. Each chunk is stored in a keyword index, a vector index of embeddings, or both.
Retrieval. The user's question, sometimes rewritten first, is used to fetch the top candidates.
Reranking. A slower, more accurate model, often a cross-encoder that reads question and passage together, reorders the candidates and keeps the best few.
Generation. The selected passages are placed in the prompt with instructions to answer from them and to cite them.

Each step introduces choices that change results, which is why two RAG systems built on the same model can behave very differently. As our article on reproducibility in NLP argues, chunk sizes, embedding models, index settings and prompts all need reporting.

Three ways to retrieve

Sparse retrieval

Classic information retrieval represents documents by the words they contain. BM25, formalised by Stephen Robertson and colleagues in the probabilistic relevance framework, scores a passage by how often the query terms occur in it, weighted by how rare those terms are across the collection and normalised by document length. It needs no training, is fast, and excels at exact matches such as codes, names and rare terms.

Dense retrieval

Dense retrievers encode questions and passages into vectors with a neural network and compare them by similarity. Dense Passage Retrieval (Karpukhin and colleagues, EMNLP 2020), the retriever behind the original RAG model, trained two BERT encoders on question–passage pairs and outperformed BM25 on several open-domain question answering benchmarks. Dense methods capture paraphrase ("car" for "automobile") but can miss exact strings and degrade on domains unlike their training data. The BEIR benchmark (Thakur and colleagues, NeurIPS 2021 Datasets and Benchmarks) tested retrievers zero-shot on 18 datasets and found BM25 a robust baseline that many dense models of the time failed to beat outside their training domain.

Late interaction

ColBERT, introduced by Omar Khattab and Matei Zaharia at Stanford (SIGIR 2020), sits between the two. Instead of compressing a passage into one vector, it keeps one embedding per token and scores a passage by matching each query token to its most similar passage token, then summing these maxima. Passage embeddings are computed in advance, so search stays fast, while the token-level matching keeps much of the precision of models that read query and passage together. ColBERTv2 (Santhanam and colleagues, NAACL 2022) reduced the storage cost with compressed residual representations. Khattab, Zaharia and colleagues later built DSPy (2023), a framework that treats retrieval and prompting steps as modules whose prompts and examples can be optimised against a metric rather than tuned by hand.

In practice, many systems run sparse and dense retrieval in parallel and merge the two ranked lists, for example with reciprocal rank fusion (Cormack, Clarke and Büttcher, SIGIR 2009), which needs no score calibration.

Where RAG goes wrong

Chunking. A table split from its header, or an answer split across two chunks, cannot be retrieved as a whole. Splitting on document structure (sections, paragraphs) with some overlap usually works better than fixed character counts.
Missing or stale documents. No retriever can find what was never indexed. A system that cannot say "the documents do not cover this" will fall back on the model's own memory.
Position in the context. In "Lost in the Middle" (TACL 2024), Nelson Liu and colleagues showed that models used relevant information best when it appeared at the beginning or end of a long context, and noticeably worse when it sat in the middle. Passing twenty passages instead of five is not free.
Unfaithful answers. The generator may ignore the passages, blend them with prior knowledge, or cite a passage that does not support the claim. Retrieval reduces unsupported statements; it does not eliminate them.
Language mismatch. A question in one language over documents in another needs a multilingual embedding model or translation, and quality often drops for less-resourced languages, a gap discussed in our article on low-resource languages.

How to evaluate a RAG system

The single most useful habit is to evaluate retrieval and generation separately. An end-to-end accuracy score cannot tell you whether a wrong answer came from a missing passage or from a model that misread a correct one.

Retrieval

Build a test set of questions with the passages that answer them, then measure recall@k (is a relevant passage among the top k?), mean reciprocal rank (how high is the first relevant passage?) and nDCG (does the ranking put the most relevant passages first?). These are standard information retrieval measures; our guide to evaluating NLP models covers the general cautions about test sets, splits and contamination that apply here too.

Generation

Two questions matter. Is the answer correct, judged against a reference? And is it faithful, meaning every claim is supported by the retrieved passages? The KILT benchmark (Petroni and colleagues, NAACL 2021) made this distinction concrete by scoring answers only when the system also retrieved the right supporting page. Tools such as Ragas (Es and colleagues, 2023) use a language model as judge to estimate faithfulness, answer relevance and context relevance without human labels. Such judges are convenient but have their own biases; check a sample of their verdicts against human annotation before trusting them at scale.

A minimal routine

Collect 50 to 200 real questions, including some the documents cannot answer.
Label the supporting passages for each.
Measure retrieval alone, and compare BM25, a dense model and a hybrid before tuning anything else.
Measure answers for correctness and faithfulness, and check that unanswerable questions get a refusal.
Rerun the same set after every change to chunking, models or prompts, and keep the results with the configuration that produced them.

Why it matters beyond accuracy

Retrieval changes what a system can be held to. Answers can cite their sources, documents can be updated or withdrawn without retraining, and access controls can be applied at the index. An index of internal documents may also contain personal data, which a model quoting passages will reproduce. The concerns raised in our article on memorisation in language models do not disappear with RAG; they move to the document store.

Sources

Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Lewis, Yih, Rocktäschel, Riedel and Kiela, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", NeurIPS 2020.
Khattab and Zaharia, "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT", SIGIR 2020.
Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni and Liang, "Lost in the Middle: How Language Models Use Long Contexts", Transactions of the ACL, 2024.
Karpukhin and colleagues, "Dense Passage Retrieval for Open-Domain Question Answering", EMNLP 2020; Thakur and colleagues, "BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models", NeurIPS 2021.
Petroni and colleagues, "KILT: a Benchmark for Knowledge Intensive Language Tasks", NAACL 2021; Es, James, Espinosa-Anke and Schockaert, "Ragas: Automated Evaluation of Retrieval Augmented Generation", arXiv:2309.15217, 2023.