Retrieval-augmented generation (RAG)
Retrieval-augmented generation answers a question by first searching a document collection, then giving the retrieved passages to a language model as the material for its reply. The model is grounded in those passages rather than in what it memorised, so answers can cite a source and change when the documents do.
How does a RAG pipeline work?
- Ingestion: documents are split into passages sized to be retrieved and read independently.
- Indexing: each passage is embedded as a vector and stored, often alongside a keyword index.
- Retrieval: the question is embedded and the closest passages are returned, typically with a keyword search combined in.
- Generation: the passages are supplied to the model with an instruction to answer from them and to say when they are insufficient.
- Attribution: the answer is returned with the passages it drew on, so a reader can verify it.
Retrieval quality sets the ceiling on answer quality. A model given the wrong three passages will produce a fluent answer from the wrong three passages, and no amount of prompt engineering downstream recovers a fact that was never retrieved.
Where do RAG systems usually fail?
- Chunking that splits a table from its header, or a clause from the definition it depends on.
- Pure vector search on queries where an exact term matters, such as a part number or a policy code.
- Stale indexes: the document was updated, the index was not, and the answer is confidently out of date.
- No abstention path, so the model answers from insufficient passages instead of saying the collection does not cover the question.
- Access control applied at the interface rather than at retrieval, which lets the index leak what the UI hides.
Access control belongs at retrieval time. If a passage a user may not read can enter the model's context, the answer can restate it, and the permission check in front of the chat window has been bypassed.
RAG or fine-tuning?
| RAG | Fine-tuning | |
|---|---|---|
| Adds new facts | Yes, immediately on re-index | Slowly, by retraining |
| Can cite a source | Yes | No |
| Handles frequent updates | Well | Poorly |
| Teaches format and tone | Weakly | Well |
| Cost of a change | Re-index affected documents | A training run |
They solve different problems and are often combined. Retrieval supplies facts that change; fine-tuning shapes how the model writes and what conventions it follows. A knowledge base is a retrieval problem, and treating it as a training problem is the most expensive way to get an answer that cannot cite anything.
Frequently asked questions
- What is retrieval-augmented generation?
- Retrieval-augmented generation answers a question by searching a document collection first and giving the retrieved passages to a language model as the material for its reply. The model is grounded in those passages rather than in memorised training data, so answers can cite sources and update when documents do.
- Does RAG stop a model from making things up?
- It reduces it substantially by giving the model the material to answer from and a way to say the collection does not cover the question. It does not eliminate it: a model can still misread a passage or fill a gap, which is why answers should be returned with the passages they used.
- Is RAG better than fine-tuning?
- They solve different problems. RAG supplies facts that change and can cite them; fine-tuning teaches format, tone and conventions. A knowledge base that updates weekly is a retrieval problem, and fine-tuning it is an expensive way to get answers that cannot cite anything.
- Why do RAG systems return wrong answers?
- Almost always because retrieval returned the wrong passages: chunking split a table from its header, vector search missed an exact term like a part number, or the index was stale. Retrieval quality is the ceiling on answer quality, and prompt changes downstream cannot recover a fact never retrieved.
- How does access control work in RAG?
- It has to be enforced at retrieval, filtering the index by the requesting user's permissions before passages reach the model. Applying it only at the interface lets the model restate content the user cannot open, which converts a permissions boundary into a formatting preference.
- Can RAG run entirely on-premise?
- Yes. The embedding model, the vector store, the retrieval layer and the generation model can all run on local hardware, which is what an on-premise or air-gapped knowledge assistant requires. Nothing in the pattern depends on a hosted service.