Why chunk at all?
Because a single vector cannot stand for a seventy-page contract. Embedding compresses text into a fixed-length row of numbers: if the text is short, that is precise; if it is long, the point gets lost in the average. That mechanism is what the embeddings and vector search piece explains at length.
The measured difference comes from a large provider's own study, across four customer datasets:
| Approach | Recall@50 |
|---|---|
| One vector for the whole document | 28.2 |
| Chunked (512 tokens, 25% overlap) | 45.7 |
(This is a separate sub-experiment of that measurement; the size and overlap tables further down come from a different sub-experiment, so their absolute values cannot be compared with this one.)
This is the strongest and simplest argument there is: 17.5 points from chunking alone.
How big should a chunk be?
The same measurement worked through the sizes as well:
| Chunk size | Recall@50 |
|---|---|
| 512 tokens | 42.4 |
| 1,024 tokens | 37.5 |
| 4,096 tokens | 36.4 |
| 8,191 tokens | 34.9 |
Monotonic decline. From 512 to 8,191 the drop is 7.5 points. The reason is simple: the relevant sentence gets diluted inside the chunk. If one paragraph in an 8,000-token block answers the question, the block's vector is mostly about the other 7,800 tokens.
But here is the caveat, without which that number misleads. A 2025 study across several datasets found that there is no universal optimum:
| Chunk size | When it is best |
|---|---|
| 64–128 tokens | tight, fact-based questions ("what is the notice period?") |
| 512–1,024 tokens | where wider context is needed ("why was the contract amended?") |
And the embedding model has a say too: in that study one model benefited from larger chunks on long documents, while the other did better with smaller ones when searching for names and identifiers.
So the responsible way to put it: 512 tokens is a good starting point, not an answer. The answer is whatever you measure on your own questions.
Small chunks, better precision
There is a second measurement that looks at the same thing from another angle — and it matters, because it does not measure the same thing.
| Strategy | Size / overlap | Recall | Precision |
|---|---|---|---|
| Recursive | 800 / 400 | 85.4% | 1.5% |
| Recursive | 400 / 0 | 89.5% | 3.6% |
| Recursive | 200 / 0 | 88.1% | 7.0% |
| Cluster-based semantic | 200 / 0 | 87.3% | 8.0% |
| LLM-driven chunking | — | 91.9% | 3.9% |
Recall measures whether you found the answer. Precision measures how much useless text you dragged along with it.
There is barely three points between the recall of the 800/400 and the 200/0 setting — but precision differs by four and a half times. In other words, big chunks find the answer just as often, while putting four times as much noise in front of the model. And noise is not neutral: it degrades the answer, and you pay for it in tokens.
Two caveats that have to be said out loud. First: the variance in this measurement is enormous, between ±25 and ±37 points — so a few points of difference between averages does not tell you much on its own. Second: the measurement was run by a company that sells a vector database, albeit with an open methodology and open code.
On overlap, three sources say three things
This is the point where most guides sound more confident than the evidence warrants. Here is what they say:
- A large provider's measurement: 25% overlap brought +3 points of recall over a plain cut.
- The chunking study above: overlap hurt precision — more redundant tokens.
- A 2026 preprint: overlap raises the cost of indexing with no measurable gain in quality.
The honest summary: overlap gives little, and it definitely costs something. If your documents are short, self-contained paragraphs, you can probably drop it. If it is continuous prose where a thought can run past a boundary, keep it — but 10–25% is plenty.
The boundary you cut on
From the same measurement, and this is the cheapest improvement in the whole business:
| Strategy | Recall@50 |
|---|---|
| Cut on a token boundary | 40.9 |
| Preserve sentence boundaries | 42.4 |
| 10% overlap | 43.1 |
| 25% overlap | 43.9 |
Preserving sentence boundaries is one and a half points, for free. A sentence cut in half yields two meaningless fragments, and both get a bad vector.
The factory default is not good
This is the most practical finding in the article, because most RAG systems run on factory settings.
| Tool | Default chunk size / overlap |
|---|---|
| One widely used framework | 4,000 characters / 200 |
| Another one | 1,024 tokens / 20 |
| A large provider's assistant feature | 800 tokens / 400 |
Of that last one, the independent measurement says word for word that it gives "below-average recall and the worst score on every other metric".
Not one of the factory defaults comes close to what the measurements suggest. This is the half hour of work that pays off most in a RAG project.
Tables, lists, code, PDFs
Here is the real work, and here is where it pays back best.
The best measurement on this was run on 80 difficult financial reports — 4 to 549 pages, 147 pages on average, 11,787 pages in total. The comparison was between fixed-size chunking and document-structure-aware chunking, where the system recognises headings, paragraphs, list items and tables:
| Strategy | Page-level accuracy |
|---|---|
| Fixed 128 tokens | 72.3% |
| Fixed 256 tokens | 73.1% |
| Fixed 512 tokens | 68.1% |
| Structure-aware | 84.4% |
Eleven to twelve points more, even against the best fixed size — and answer accuracy rose from 48.2% to 51.8–53.2% as well.
And the most important side number: it did all that with close to half as many chunks — 62,529 against 112,155. That means close to half the storage, close to half the embedding cost, less noise. This is where the work invested in chunking measurably pays for itself.
Four concrete rules for the details, from primary documentation:
1. TABLES: REPEAT THE HEADER IN EVERY CHUNK
If you cut a 200-row table into four chunks and
the header is only in the first one, chunks 2-4
are a pile of number columns with no meaning.
Good tools have this switched on by default.
2. NEVER MERGE A TABLE WITH OTHER TEXT
It is a unit of its own. If it does not fit,
chunk it further as a table, not as text.
3. LISTS: DECIDE WHETHER IT IS ONE UNIT
Tool defaults are the OPPOSITE of each other:
one keeps the list together, the other cuts it.
Check which one you are working with.
4. CODE: CUT ON LINES, NOT ON CHARACTERS
A line of code is a unit; character 400 is not.
There are syntax-aware splitters made for this.
The first rule is the most common silent failure. Embedding a header-less table fragment effectively encodes noise, and retrieval will never find it.
When a chunk loses its context
There is a technique that solves exactly this problem: a short generated sentence is placed in front of every chunk saying where it sits in the document — before you embed it. "This excerpt is from the section of the 2024 annual report covering third-quarter revenue."
The vendor's measured result (retrieval failure rate, top 20 results):
| Setting | Failure rate | Improvement |
|---|---|---|
| Baseline | 5.7% | — |
| + context sentence before the chunk | 3.7% | −35% |
| + keyword search as well | 2.9% | −49% |
| + re-ranking | 1.9% | −67% |
The cost, according to the vendor's own figures: $1.02 per million document tokens, a one-off item, with caching.
(Vendor measurement, with their own model. The method, however, is reproducible with any model, and its logic can be checked.)
Notice, though, that the single biggest step here was the context sentence — from 5.7 to 3.7 — while re-ranking took away a third of the failures that were left. That is the subject of the hybrid search article, in more depth.
The how-to
1. CHECK THE FACTORY DEFAULT
This is step one, and it pays the most.
If it is 800, 1024 or 4000, change it.
2. START AT 512 TOKENS, ON SENTENCE BOUNDARIES
A good start. Not an answer - a start.
3. OVERLAP 10-25%, OR NONE AT ALL
Keep it for continuous prose, drop it for
self-contained paragraphs. Never more.
4. USE THE STRUCTURE, NOT THE CHARACTER COUNT
Heading, paragraph, list item, table. Measured:
+12 points, with CLOSE TO HALF as many chunks.
5. REPEAT THE HEADER ON TABLES
In every chunk. This is not optional.
6. PUT THE DOCUMENT TITLE IN EVERY CHUNK
The cheapest context you can give.
7. MEASURE TWO THINGS SEPARATELY
Did it find it (recall) AND how much noise came
with it (precision). Not the same question.
8. TEST ON YOUR OWN QUESTIONS
There is no universal optimum. Twenty real
questions beat any blog post.
The seventh point is the one most people skip. If you only measure whether it found the answer, big chunks will look good — while putting four times as much noise in front of the model.
Common misconceptions
- "There is an optimal chunk size." There is not. On the measurement it is 64–128 tokens for fact-based questions and 512–1,024 where context is needed — and the embedding model has a say too.
- "Bigger chunks give more context, so they are better." Recall declines monotonically with size: 42.4 at 512, 34.9 at 8,191. The relevant part gets diluted.
- "More overlap, safer hit." Three measurements say three things. The honest summary: it gives little, and it definitely costs something.
- "Semantic chunking is always better." On real data, fixed-size chunking is often just as good or better — and much cheaper.
- "A better embedding model will fix it." A 2026 measurement found that larger models score higher but are just as sensitive to bad chunking.
- "Chunking is a one-off job." When a new document type arrives (different PDF structure, different table format), you have to look at the whole thing again.
What to take away
Chunking is the most underrated part of RAG: it is invisible, there is no demo for it, and it is still where the quality of the system is decided. Half the failures — the answer not making the results, or making them incomplete — trace back to chunking.
Three numbers worth remembering: +17.5 points from chunking at all; +12 points from close to half as many chunks if you use the document's structure; and a four-and-a-half-fold difference in precision between small and large chunks at practically identical recall.
And the fastest win available to you today: go and look at the default chunk size in your system. If it is 800 or more, one setting will improve things more than a change of model. How you tell that a finished system answers well is what the RAG evaluation post walks through, and for the storage side there is the vector database primer.