A language model cannot search a hundred-page manual in one go. RAG systems therefore cut the text into pieces first, and hand the model the pieces that match the question best. Chunking is the step where it is decided what goes into a piece — and with that, what the system will ever be able to find.
The effect is measurable and large. In one measurement, simply splitting the long document at all lifts retrieval quality from 28.2 to 45.7 — 17.5 points, with nothing else changed. Splitting along the structure (headings, sections) adds another 12 points on top of that, and does it with roughly half as many pieces.
The most common mistake is keeping the factory default. The most widely used tools work with pieces of 800 and 1,024 tokens, or of 4,000 characters — and in an independent measurement the 800 setting finished at the bottom of the field. Bigger pieces are monotonically worse: more text gets in, and less of it is relevant.
Splitting strategies, overlap and how to measure any of it are covered by Preparing and Chunking Your Documents in detail. Directly related terms: chunk overlap, embedding and retrieval.