Sentence Splitting
Also called Sentence Segmentation
Cutting text at sentence boundaries so chunks don’t start or end mid-thought.
Think of it like
Tearing a page along the perforations instead of ripping through the middle of a line.
Example
A chunker uses sentence splitting so "The dose is 5mg. Do not exceed" never gets severed between "5mg" and "Do not exceed".
How it actually works
Naively slicing every N characters can cut a sentence in half, garbling both pieces. Sentence splitting detects boundaries — trickier than it sounds, thanks to "Dr.", "3.14", and abbreviations — and packs whole sentences into chunks. It is the humble preprocessing step that keeps retrieved passages readable and meaning intact.
For product teams
Unglamorous but real: clean boundaries mean chunks the model can actually use.
For engineers
Boundary detection (rules or ML) to avoid mid-sentence cuts; edge cases are abbreviations, decimals, and quotes.
Related
- Chunking — A building block of chunking.
- Semantic Chunking — Feeds into semantic chunking.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome