If a RAG system is giving bad answers, chunking is the first place to look - more often than the embedding model, the vector database, or the LLM itself. A chunk is the unit your system retrieves and hands to the model, so if the boundaries are wrong, everything downstream inherits that mistake.
Fixed-size chunking: simple, and often good enough
The most basic approach: cut the text every N characters (or tokens), full stop, no regard for what's actually at that boundary. It's trivial to implement and completely predictable in size, which matters for cost and latency. Its weakness is exactly what you'd expect - it will happily slice a sentence, a table row, or a code block in half, because it has no idea those things exist.
Recursive chunking: the practical default
Recursive character splitting tries to respect the document's structure before falling back to a hard cut: split on paragraph breaks first, then sentences, then words, only cutting mid-word as an absolute last resort. This is the default most teams reach for, and for good reason - it's a large quality improvement over fixed-size splitting for almost no extra cost.
# Recursive splitting, framework-agnostic shape
def recursive_split(text, chunk_size=1000, overlap=200):
separators = ["\n\n", "\n", ". ", " "] # tried in order
# Splits on the first separator that produces chunks
# close to chunk_size; falls back to the next one, and
# finally to a hard character cut if none fit.
...
Semantic chunking: split where the topic actually changes
Semantic chunking goes a step further: instead of splitting on structural markers like paragraph breaks, it embeds sentences and looks for the points where meaning shifts noticeably from one sentence to the next, and splits there. The result is chunks that are more likely to be topically coherent - a chunk about "shipping times" doesn't get contaminated with three unrelated sentences about "payment methods" that happened to sit in the same paragraph. It's more expensive to compute (every sentence gets embedded during chunking, not just during the final chunk) and it's usually only worth the cost once fixed or recursive splitting has visibly hit a ceiling on your specific documents.
| Strategy | How it decides where to cut | Best for |
|---|---|---|
| Fixed-size | Every N characters, no exceptions | Quick prototypes, uniform plain text |
| Recursive | Paragraph → sentence → word, in that order | The default for most real documents |
| Semantic | Where embedded meaning shifts between sentences | Long, topic-dense documents where recursive still underperforms |
Chunk size and overlap: the tradeoff underneath all three
Chunk overlap - letting consecutive chunks share a slice of text at the boundary - softens the "cut off mid-idea" problem, so an idea that straddles a chunk boundary still appears intact in at least one of the two chunks. A common starting point is 500-1000 characters per chunk with 10-20% overlap, but treat that as a starting point, not a rule: the right numbers depend heavily on your documents and the kinds of questions people actually ask.
Chunking isn't just text - metadata rides along
Every chunk should carry metadata alongside its text: source document, page number, section heading, last-updated date. This isn't optional polish - it's what lets you show a citation the user can verify, filter retrieval to only recent documents, or exclude an entire outdated file without re-indexing everything else. A RAG system with no metadata is a RAG system that can't tell you where an answer came from.