Chunking Strategies: How to Split Text for RAG

How you cut your documents into pieces shapes everything downstream in a RAG system - too big and search gets fuzzy, too small and context disappears.

Why chunk size matters this much

A RAG system can only retrieve whole chunks, not partial ones, so the size you pick decides what the model sees. Chunks too big bury the relevant sentence in irrelevant text. Chunks too small lose the surrounding information needed to make sense of what they say - 'revenue grew 3%' means little without the company name one chunk over.

Fixed-size chunking: the simple default

The most common starting point: cut text into pieces of roughly 200-500 tokens and start there. Easy to implement, reasonable as a baseline.

Overlap between chunks

Cutting text at hard boundaries risks slicing an idea in half, at the edge of two chunks. Repeating the last portion of one chunk at the start of the next reduces the odds that a detail gets orphaned at a cut point.

Semantic and structure-aware chunking

Semantic chunking looks at where the topic shifts and cuts there, grouping sentences about the same idea. Structure-aware chunking splits along natural boundaries like paragraphs or headings. Both produce more coherent chunks than a mechanical cutoff, at the cost of more processing.

Contextual Retrieval: fixing lost context

Anthropic's Contextual Retrieval technique prepends each chunk with a short, auto-generated context blurb before embedding it, so a chunk that would read as meaningless alone carries enough context to be found correctly.

EXAMPLE

A structure-aware chunk boundary: split a Markdown doc at every heading instead of every 300 characters, so each chunk stays a complete, self-contained section instead of an arbitrary slice.

🛠️ EXERCISE — TRY IT YOURSELF

Chunk a short document two different ways and compare what a query retrieves.

Take a text file of a few pages, e.g. an article or a README.
Split it into fixed-size chunks of about 300 tokens with no overlap.
Split the same file again along its natural headings or paragraphs instead.
Pick a specific question and check, for both versions, whether the chunk containing the answer would actually be retrievable on its own.

✅ SELF-CHECK

☐ Did you test both chunking variants on the same 3 test questions?
☐ Do you know which strategy had the right chunk near the top more often?
☐ Can you say in one sentence why that strategy did better?

QUICK QUIZ

What problem does Anthropic's Contextual Retrieval technique specifically address?