What is RAG, in one sentence?
The name comes from a 2020 research paper. The authors defined it as a model that ties together trained ("parametric") knowledge and an external, searchable memory to produce an answer.
In practice that is three steps, on every single question:
1. Your question becomes a search
The system finds which document fragments are
relevant. (This is "retrieval".)
2. The results go into the prompt
Not the whole document - a few selected
fragments, next to the question.
3. The model answers from that
BESIDES its own knowledge, your own text is
there too. (This is "augmented generation".)
The key is the moment right after step three: the model forgets. Its weights did not change. On the next question the search has to run again, and the fragments go in again. That difference is what the fine-tuning, RAG and prompting comparison works through in detail, and the maths behind the search is the subject of the embeddings and vector search explainer.
One conceptual correction that trips a lot of people up: the original 2020 RAG was a training recipe — the retriever and the text generator were tuned together. What people call RAG today typically involves no training at all: an off-the-shelf embedding model, an off-the-shelf language model, and a retrieval layer between them.
What RAG does not solve
This is the backbone of the article, because this is where most projects fail.
| What does it change? | When do you need it? | What does it cost? | |
|---|---|---|---|
| RAG | nothing in the model; it puts retrieved text into the prompt | when it has to know a fact that changes, or that only you have | a retrieval layer, and extra input tokens on every question |
| Fine-tuning | the model's weights | when it has to learn behaviour, format, tone | a training run, and another training run for every new fact |
| Long context window | nothing; you paste the whole corpus in | when the whole corpus fits, and does not change daily | the token price of the whole corpus on every question |
The most common mistake is that someone wants the second column and builds the first — or the other way round. When the middle column is worth it anyway is measured in the fine-tuning payback post, with numbers.
And there is a measurement. A 2023 study tested three seven-billion-parameter models on what happens when entirely new knowledge has to be learned. The task was 910 questions about events none of the models could have seen during training:
| Model | Base | + fine-tuning | + RAG |
|---|---|---|---|
| Model A | 48.1% | 50.4% | 87.5% |
| Model B | 35.3% | 21.9% | 58.5% |
| Model C | 45.6% | 51.1% | 87.6% |
Two things can be read out of it. One is that RAG helps by an order of magnitude more — 39.4 percentage points on the first model, while fine-tuning managed 2.3. The other is the second row: there fine-tuning made it 13.4 points worse. Not "helped less" — it did damage.
There is a third, even more uncomfortable detail from the same study: on another task the fine-tuned model with RAG (65.9%) came out worse than the plain base model with RAG (68.1%). The authors' summary: using the base model as the generator was consistently better than fine-tuning alone.
(Preprint, three seven-billion-parameter models — the exact figures do not carry over to today's large models. The direction does.)
How much does it help with hallucination?
There is no single number for this, and anyone who quotes you one probably has not measured it. What does exist: task-specific measurements with definitions that cannot be compared to each other. That in itself is useful information.
The best of them is a peer-reviewed study that tested 202 carefully constructed questions against three dedicated, professional legal research tools — so not home-made tinkering, but expensive, purpose-built RAG products.
The definition first, because that is the useful part. According to the study, an answer counted as hallucinated if it was either factually wrong, or poorly grounded — that is, it cites a source that exists but does not support what it claims.
That second case is the one RAG appears to solve (there is a citation!) and in fact does not.
| Tool | Accurate answer | Hallucinated | Incomplete |
|---|---|---|---|
| A | 65% | 17% | 18% |
| B | 41% | 33% | 25% |
| C | 19% | 17% | 62% |
(The three values are the study's own categories; because of rounding, the rows do not everywhere add up to exactly a hundred.)
The authors' conclusion: while RAG visibly helps, the hallucination problem persists at a significant level.
And a twist that most summaries get wrong: next to tool C's 17% hallucination rate you have to write its 62% incompleteness rate — it gave no substantive answer to nearly two thirds of the questions. Without the incompleteness rate the three numbers cannot be compared. Why a model invents things in the first place is covered by the causes of hallucination piece, step by step.
The contrary evidence: RAG does not make you safer
Plenty of people skip this part, and yet it is the most surprising finding in the field.
A peer-reviewed 2025 study tested 11 models with 5,592 harmful questions across 16 risk categories. The question was: does a model become safer by working only from retrieved documents?
| Measurement | Value |
|---|---|
| Unsafe answers from an eight-billion-parameter model without RAG | 0.3% |
| The same model with RAG | 9.2% |
| Retrieved documents that themselves contained harmful material | 5.3% |
| Share of unsafe outputs that came from SAFE documents | 81.8% |
A thirtyfold deterioration — and four fifths of that deterioration has nothing to do with the system pulling up a bad document.
The authors' explanation: safety fine-tuning teaches the model not to process certain directions. Once it is handed relevant context, however, it stops perceiving the topic as risky, and processes it.
Two further findings from the same source: models add their own internal knowledge to the retrieved documents even when the instruction forbids it; and longer context increases the vulnerability. Plus one operational consequence: safety tests that worked in a RAG-free environment stopped working once ported over — meaning your existing testing does not measure your RAG system.
Seven points where it breaks
A 2024 experience report — drawn from three real deployments, including a medical system with 4,017 documents and 1,000 questions — set up a failure taxonomy. It is worth learning, because every RAG failure fits somewhere in it:
| # | Failure point | What happens |
|---|---|---|
| 1 | Missing content | The answer is not in the documents, and the system generates anyway |
| 2 | Missed the top results | It is in there, but it does not make the top few |
| 3 | Dropped out of the context | It made the results, but consolidation threw it out |
| 4 | Not extracted | It is right there, and the model does not use it — noise or contradiction |
| 5 | Wrong format | It ignores the formatting instruction |
| 6 | Wrong specificity | Too general or too specific an answer |
| 7 | Incomplete | It leaves out something that was in the context |
The first failure point is the important one, because it is the only one no technique can fix. If your documents do not contain the answer, no chunking, no retrieval and no model will conjure it up.
And the single most important sentence in the report, one every buyer ought to know:
"Validation of a RAG system is only feasible during operation."
Which means: you cannot tell during development whether it will be any good. Not because the developer is bad, but because the system's behaviour depends on the real questions and the real documents. RAG is not a project, it is an operation. How to measure it is walked through by the RAG evaluation article, in detail.
When should you NOT build RAG?
Five cases, each from a primary source or a measurement:
1. When the whole corpus fits in the context window. This has the most concrete vendor threshold behind it, and it is unusually honest, because it argues against their own more complicated offering:
"If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt, with no RAG at all."
(The 500 pages refer to English text. The token ratio is worse for Hungarian, so the threshold sits at considerably fewer pages — the tokens and the context window explainer gives you a reference point for that.)
2. When the questions are not fact-based. Style, format, tone: that is fine-tuning, not RAG.
3. When nobody is going to maintain it. See above: validation is only feasible during operation — and a stale knowledge base is measurably worse than an empty one, as the keeping the knowledge base fresh post shows.
4. When your documents do not contain the answer. The first failure point cannot be fixed.
5. When you mean it as a security measure. On the measurement, it backfires.
To that add a useful number on how much better RAG is than plain long context in the first place. A 2024 study compared the two across nine datasets and found that on 63% of questions the two approaches gave exactly the same answer, and on 70% the score difference stayed below ten.
(Measured with mid-2024 models. The idea behind the 63% agreement carries over, the exact percentages do not.)
The cost, where you would not expect it
[Own calculation] Take a knowledge base of ten thousand documents, five pages each — roughly 25 million tokens, cut into 512-token chunks with 25% overlap, 65 thousand chunks.
The size of the raw vectors:
65,000 chunks × 1,536 dimensions × 4 bytes = 0.40 GB
The retrieval index next to that is negligible: roughly 10 MB at the same size. With metadata and raw text included, realistically we land around 1 GB.
One gigabyte. That is it. Monthly storage at the providers is measured in tenths of a dollar.
And here is the point: the monthly minimum on entry-level plans runs between $16 and $250. Which means your usage will sit orders of magnitude below the minimum. Your real cost is not the usage but the minimum fee — and that is what most calculations leave out.
The one-off cost of embedding is just as surprising: embedding those 25 million tokens costs less than a dollar.
The money is in answering. The same ten thousand questions a month cost roughly $26 to $635, depending on which model you pick. A twenty-fourfold difference, for the same work.
Three conclusions follow:
- Storage and embedding are effectively free. Anyone economising there is looking in the wrong place.
- Model choice is the real cost line. That is where the twenty-fourfold multiplier sits.
- Minimum fees make a small knowledge base disproportionately expensive. A thousand-document system costs about the same per month as a ten-thousand-document one.
The how-to
1. ASK FIRST: FACT OR BEHAVIOUR?
Fact -> RAG. Style, format, tone ->
fine-tuning. That is the decision, not the tech.
2. MEASURE HOW BIG THE CORPUS IS
Under 200 thousand tokens, paste it in whole.
RAG only above that.
3. START WITH TWENTY REAL QUESTIONS
Write down what they will ask, AND the right
answer. Without that you have no yardstick.
4. CHECK WHETHER THE ANSWER IS THERE AT ALL
The most common failure is that it is not. You
check this by hand, not with software.
5. MEASURE RETRIEVAL FIRST, NOT THE ANSWER
If the right fragment never makes the results,
the model has nothing to work with.
6. PLAN AN OPERATION, NOT A PROJECT
Validation is only feasible during operation.
Give it an owner and a monthly review.
7. SOLVE PERMISSIONS ON DAY ONE
Who may see what. Retrofitting costs far more,
and some systems cannot do it at all.
8. BUDGET THE MINIMUM FEE, NOT THE USAGE
Storage is pennies. The monthly minimum is not.
The seventh point is what most people spot too late. If the chatbot also hands back what the person asking is not allowed to see, that is not a bug but a data breach — and in several systems permission handling cannot be switched on afterwards.
Common misconceptions
- "RAG teaches the model my documents." It does not. The model's weights stay as they were. The document reaches it at the moment of the question, inside the prompt, and after the answer it is gone.
- "If there is a citation, it must be true." By the definition in the study quoted above, one half of hallucination is precisely citing a real source that does not support the claim.
- "RAG makes the system safer." In an 11-model measurement the share of unsafe answers grew thirtyfold, and four fifths of the damage came from safe documents.
- "You can test it up front." According to the experience report, validation is only feasible during operation.
- "A small knowledge base is cheaper." Storage is, but because of the monthly minimums a thousand-document system costs about as much as a ten-thousand-document one.
- "A better model will sort it out." Three of the seven failure points arise in retrieval and chunking — the model has no visibility there.
What to take away
RAG is the most effective way to get a model to answer from your facts: on the measurement it delivers an order of magnitude more than fine-tuning for new knowledge, and fine-tuning occasionally even hurts.
But three things are worth seeing realistically. It is not teaching — the model does not get smarter, only better informed, one question at a time. It does not solve everything — hallucination stayed between 17% and 33% even in professional products, and it measurably degrades safety. And it is not a project but an operation — validating it is only feasible while it runs.
So the best first step is not a technical one: write down twenty real questions and check by hand whether your documents contain the answers. If they do not, RAG will not help — and you find that out in a morning, not in three months.
Document preparation is the subject of the chunking guide, and storage is covered by the vector database primer.