A language model can write fluently about almost anything, but it cannot read your company’s latest policy document, and it knows nothing that happened after its training data was collected. Retrieval-augmented generation, usually shortened to RAG, is the standard workaround: before the model answers, a search step finds relevant documents and hands them to the model along with the question.
Where the idea came from
The name comes from a 2020 research paper, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” by Patrick Lewis and colleagues, presented at the NeurIPS conference. The authors described models that combine two kinds of memory. Parametric memory is what a pre-trained language model has absorbed in its weights. Non-parametric memory is an external store the model can search, which in the paper was a dense vector index of Wikipedia.
The paper’s motivation is still the main argument for RAG today. The authors noted that language models struggle to access and precisely manipulate stored knowledge, and that giving a model explicit, searchable memory could help with two open problems: showing where an answer came from, and updating a model’s knowledge without retraining it.
What happens when you ask a question
A typical modern system works in four steps.
First, documents are prepared in advance. They are split into short passages, and each passage is converted into an embedding, a list of numbers that captures its meaning. The embeddings are stored in a database built for similarity search.
Second, when a question arrives, it is converted the same way, and the system retrieves the passages whose meaning sits closest.
Third, those passages are placed in the prompt with the question and an instruction to answer from them.
Fourth, the model writes its answer, ideally with references to the passages it used so a reader can check them.
The effect is something like an open-book exam. The model still does the writing, but it has the relevant pages in front of it.
What RAG fixes
Because the knowledge lives in a document store, changing it is cheap: update a file and the next answer reflects the change. Private material such as manuals, contracts or support histories can be searched without being built into the model. And when answers come with citations, a reader can verify them, which is harder with a model answering from memory.
What RAG does not fix
RAG changes where the information comes from, not how the model behaves. Several failure points remain.
Retrieval can miss. If the search step returns the wrong passages, the model may answer confidently from irrelevant text, and many apparent model failures are really search failures.
The model can still stray. Having the right passage in the prompt does not guarantee the answer stays faithful to it, so the underlying issue discussed in our explainer on AI hallucinations is reduced rather than removed.
The documents themselves matter. Outdated, contradictory or badly organized sources produce answers with the same flaws.
Access control is easy to get wrong. A system that searches everything can surface documents a particular user should not see unless permissions are enforced at retrieval time.
RAG compared with fine-tuning
People often ask whether RAG replaces fine-tuning, which adjusts the model’s weights on new examples. They solve different problems. Fine-tuning is better for teaching a model a style, a format or a narrow skill. RAG is better for facts that change or that must stay private. Many products use both.
How to judge a RAG product
A few questions separate careful systems from careless ones. Does it show which passages it used? Does it say so when it finds nothing relevant, instead of guessing? Does it respect each user’s permissions? And has the team tested it on real questions with known answers, then retested after changing the documents or the model? An assistant that can answer yes to those is far more trustworthy than one that merely claims to know your data. RAG also pairs naturally with AI agents, which can treat a document search as one of the tools they call.
