Chunk Size
Also called Chunk Length
How big each piece of a document is when you split it for retrieval — a surprisingly high-stakes knob.
Think of it like
Cutting a sandwich: too big to fit the box, too small and the filling falls out.
Example
A team switches from 1000-token chunks to 300-token ones and suddenly answers get sharper because passages are more focused.
How it actually works
Chunk size trades context for precision. Large chunks keep surrounding meaning but dilute the embedding and waste tokens; small chunks are precise but can strand a fact from the sentence that explains it. There is no universal best — it depends on content and query style — so it is one of the first things worth tuning, often with a bit of overlap between chunks.
For product teams
One of the cheapest levers on answer quality; worth experimenting with early.
For engineers
Token/char span per chunk; governs embedding specificity vs. context retention, usually tuned with overlap.
Related
- Chunking — A key parameter of chunking.
- Retrieval Quality — Directly affects retrieval quality.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome