First: do you actually need to chunk?
Many people chunk when there is no need. Today's context window is typically around a million tokens — that is a few hundred pages. A forty-page contract fits comfortably, and processed in one piece it gives a more accurate answer than chunked, because the model can see the connections between sections.
Three signs say you do need to chunk after all:
- It does not fit. A multi-hundred-page documentation set, a year of correspondence, a complete body of legislation.
- It fits, but quality drops. Accuracy starts falling well before the limit: in a very long text the model uses information in the middle less reliably than information at the beginning or the end. If the hits are missing from the middle of the material, that is the symptom.
- You are going to ask the same thing many times. If the entire file passes through on every question, the bill grows quadratically.
The method: split, extract, combine
Three steps, and most errors are decided in the second one.
1. Split. Cut the material into manageable chunks.
2. Extract. You run the same question on every chunk, and you do not ask for a summary but for the facts relevant to the question.
3. Combine. You hand over the extracts together, and the final answer is built from those.
What has to be decided in advance
What is the final question? This is not a formality. In the second step you have to pull out of every chunk what the final question needs — and that looks different per chunk if you do not know in advance what you are looking for.
If the final question is "what obligations do we have and by when", then the extract is not "what is this section about" but: obligation, deadline, owner, clause number. Ask for a summary and the next step will have nothing to work from — the summary threw away precisely the details.
Where to cut?
Along the structure of the document, not at a fixed character count. Chapter, section, agenda item, date. If one thought gets cut in two across two chunks, neither chunk will yield a good extract.
Two practical tricks:
- Overlap them. Neighbouring chunks should run a few paragraphs into each other, so a sentence on the boundary survives in both.
- Every chunk should carry where it came from. Chapter number, page number, date. Without it you will not be able to trace back at the end where a claim came from — and that is exactly why you are doing all this.
The same logic sits behind search-based systems, where we call it chunking. Embeddings and Vector Search walks around the same pitfall in more detail.
The extraction prompt that runs on every chunk
This step is good when it is boring and consistent. The same prompt, the same format, on every chunk:
<task>
From the extract below, collect the obligations.
If there are none in it, write: NONE.
</task>
<extract source="chapter 3, pages 12-18">
[the text of the chunk]
</extract>
<format>
One item per line: obligation | deadline | owner | clause number
Only what appears in the extract. Do not fill in gaps.
</format>
The NONE matters: without it the model feels it has to find something, and it will.
Combining, where the point gets lost
This is the step most often rushed. Do not just paste the extracts together — say what to do with them:
From the extracts below, produce a consolidated list. Merge identical obligations, and mark which clauses they come from. Where deadlines contradict each other, do not choose — list both, with sources.
The last sentence is what made the whole exercise worth it. A contradiction is the most valuable find in a long file, and if you do not ask for it separately, the model smooths it over — it picks one, and the other disappears.
When you need something better than this
If you regularly ask many questions of the same material, the chunking runs again every time, and that is waste. This is the territory of context engineering: at that point the thing to perfect is not the prompt but putting a search in front of it, so only the relevant part goes through. Fine-Tuning, RAG or a Better Prompt? tells you where the boundary runs.
What to take away
The final question first, then the chunking. The extract is not a summary: from every chunk you pull what the final answer needs, with the source marked. And when combining, ask for contradictions explicitly — that one sentence is what saves the point.