Four metrics, and what they actually measure
The established RAG metrics answer four separate questions. Keeping them apart is worth the effort, because most of the disappointment comes from trying to squeeze all four into a single number:
| Metric | What it measures | Where the error is created |
|---|---|---|
| Groundedness | Whether the claims in the answer follow from the retrieved text | at answering |
| Answer relevance | Whether the answer addresses the question asked | at answering |
| Context precision | How much of what was retrieved is relevant | at retrieval |
| Context recall | Whether the needed information made it in at all | at retrieval |
The top two rows are about the model, the bottom two about retrieval. If you do not measure them separately, a bad number will not tell you what to fix.
The diagnostic table
This is the most practical part of the article. Four base situations, each calling for a different intervention:
| Did the right text make it in? | Did the model use it? | Diagnosis | What to fix |
|---|---|---|---|
| No | — | The answer is not in the retrieved passages at all | chunking, index, retrieval, number of results |
| Yes | No | It was right there and went unused | prompt, context length, ranking |
| Yes | Yes, but invents on top | It used the source and added fiction | prompt, temperature, model |
| Yes | No — it answered from its own knowledge | Good answer, just not from the source | citation requirement, forcing the source |
A single aggregate number — "groundedness: 0.82" — tells none of these four apart. That is why you need two measurements, not one.
How believable are these numbers?
Here comes the most important material in the article, and it is uncomfortable.
The first measurement was built on the official assessment data of a public evaluation campaign: 45 submitted systems, 36 topics, 6,742 manual annotations. The question was whether a given claim is supported by the document cited for it.
| Comparison | Agreement (corrected for chance) |
|---|---|
| Judge model vs. human assessor | 0.29 |
| Independent human vs. human assessor | −0.03 |
The second row is the point. The minus sign means the two humans agreed worse than chance would have. So the problem is not the machine: judging partial support is not consistent between humans either.
There is good news from the same source, though, and it gives the practical answer. At system level — on the question of which configuration is better — the judge model and the human correlated strongly (above 0.79).
So the usable rule is this: a judge model is good for ranking, not for judging an individual answer. Use it to decide whether configuration A or configuration B is better. Do not use it to tell you whether this particular answer is correct.
And one systematic bias worth knowing about: in the same measurement the human assessors leaned towards the "not supported" label (47%), the model towards "partially supported" (49-51%). The judge model is the more lenient of the two — meaning you get a prettier number than a human would give you.
The second measurement is harsher still: 21 judge models from nine providers, across three benchmarks, with some 541,000 judgements. Agreement with humans, corrected for chance, ranged between 0.376 and 0.511.
And here is the key methodological finding:
Plain agreement counting overestimates chance-corrected agreement by between 33.8 and 41.3 percentage points; averaged over the 21 models, by 38.6 percentage points.
In plain terms: anyone who says "our judge agrees with humans 85% of the time" is overstating reality by nearly forty points on average.
A counter-intuitive benchmark on company data
May 2026 brought a benchmark built specifically on internal company knowledge rather than web text. The dataset is roughly half a million documents from nine sources: chat messages, email, a task tracker, a document store, tickets, meeting minutes. The evaluation runs over 500 questions in ten categories.
The result:
| Method | Correctness | Completeness | Document recall |
|---|---|---|---|
| Keyword search | 68.8% | 56.0% | 68.4% |
| Embedding search | 51.4% | 42.9% | 46.0% |
| Command-line search agent | 60.6% | 61.1% | 55.8% |
Good old keyword search beat embedding search by 17 points. On company documents.
The lesson is not that vector search is bad — it is that company data is substantially different from the public benchmarks most recommendations are built on. This class of error is what the hybrid search piece takes apart, measurement by measurement.
How many test cases do you need?
There is a concrete primary recommendation for this:
"A minimum of 100 question-answer pairs is required. For larger domains we suggest 150."
The questions should be self-contained, and the answers should be validated by an expert — especially the ones a model generated in the first place.
[Own calculation] It is worth adding what that sample size can actually detect. For a binary (passed / failed) measurement the uncertainty is roughly:
margin of error ≈ 1.96 × √( p(1−p) / n )
So with 100 test cases, at a result around 75%, the 95% margin of error is ±8.5 percentage points. Which means:
| Test cases | Detectable difference |
|---|---|
| 20 | ~±19 points — order-of-magnitude changes only |
| 100 | ~±8.5 points |
| 500 | ~±3.8 points |
| 1,200 | ~±2.5 points |
(This is the confidence interval of a single proportion. If you compare two configurations on the same test set, the uncertainty of the paired comparison is smaller than this — but the order of magnitude holds.)
On twenty test cases you cannot detect a five-point improvement. That is not pedantry, it is arithmetic — and most "it feels better to me" arguments hinge on it.
One methodological trap comes with this: if the questions arrive in clusters (ten questions from the same document, say), they are not independent, and the true margin of error can be up to three times what you calculated. In that case you have to compute at cluster level.
The noise that helps and the near-miss that kills
There is a peer-reviewed measurement that shows two apparently contradictory things in the same experiment — and both of them change how you should measure.
The first, counter-intuitive result: adding completely random, irrelevant documents to the context improved accuracy, by as much as 35%.
The second, and far more important: a single distracting document — one that is on topic but carries the wrong answer — degraded accuracy by 25%, and in some cases by 67%.
Position matters too: in the same measurement, the correct document placed near the question gave an accuracy of 0.37; placed in the middle, 0.23.
Three consequences for measurement:
- Context precision on its own is a misleading metric. According to the measurement, irrelevant noise does no harm — the near-miss result does.
- Counting how many relevant results you have is not enough. What you need to measure is whether there is a deceptively wrong one among them.
- Order matters. Two systems with the same set of results in a different order give measurably different outcomes.
The minimum setup
1. TWENTY QUESTIONS FIRST, A HUNDRED TO DECIDE
Twenty is enough to see whether it works at all.
Comparing configurations takes a hundred.
2. WRITE DOWN THREE THINGS PER QUESTION
a) the question as a colleague would ask it
b) the correct answer
c) WHICH document it has to come from
3. THE THIRD ITEM IS THE IMPORTANT ONE
It lets you measure retrieval and answer apart.
Without it you only get one lumped number.
4. MEASURE RETRIEVAL FIRST
Did the right document make it into the results?
If not, tuning the model is pointless.
5. USE THE JUDGE MODEL FOR RANKING
"Is A or B the better setup?" - good for that.
"Is this answer correct?" - not for that.
6. REVIEW TWENTY ANSWERS BY HAND EVERY MONTH
Not for an absolute verdict: you measure DRIFT.
The model is more lenient than a human - the
direction of that gap stays usable even when
humans do not agree on the edge cases either.
7. MEASURE THE DISTRACTOR, NOT THE NOISE
An irrelevant result does no harm. An on-topic
one carrying wrong information does.
8. CHECK THE CONFIDENCE BEFORE CELEBRATING
At 100 test cases it is ±8.5 points. That 74
to 78 improvement may never have happened.
Sub-item c) of the second step is the one almost everyone skips — and without it your measurement gives you one vague number instead of telling you where the fault sits.
Common misconceptions
- "The judge model tells you whether an answer is good." On individual answers its agreement with humans was 0.29 — while two humans agreed at −0.03 on the same data. For ranking, though, it works.
- "Our judge agrees with humans 85% of the time." Plain agreement counting overestimates by 38.6 percentage points on average. The real, corrected value is far lower.
- "One good metric is enough." Groundedness on its own does not separate "it never found it" from "it found it and did not use it".
- "Vector search is better than keyword search." In a 2026 benchmark on half a million company documents, keyword search was 17 points better.
- "Twenty test cases is enough." At twenty the margin of error is around ±19 points. A five-point improvement cannot be detected there.
- "Maximise the share of relevant results." The measurement says random noise helped, while an on-topic wrong result cost as much as 67%.
What to take away
RAG measurement is hard not because there are no metrics — it is hard because the reliability of the metrics is itself a measurement question, and a good share of the numbers are more optimistic than reality.
Two things work anyway. Measure retrieval and answer separately — the diagnosis depends on that, not on a more accurate metric. And use the judge model for ranking, not for verdicts: that is what it is demonstrably good for.
And the most useful thing you can do today is a table with three columns: question, correct answer, and which document it has to come from. A hundred rows. That file is worth more than any evaluation framework — because without it none of them has anything to measure. The full picture is set out in the RAG explainer pillar piece, and planning a rollout is where the company knowledge chatbot article comes in.