Chunking Strategies

When a document is split into chunks for retrieval-augmented generation (RAG) or context window management, the chunk boundaries determine what the AI can see as a unit. A fixed-size split every 500 tokens treats the document like a paper shredder — it cuts wherever the token count falls, regardless of whether that's in the middle of a sentence, an argument, or a conditional clause. The AI then receives a chunk that starts mid-thought and ends mid-thought, with no way to recover the missing context from either side.

Semantic chunking — which leverages the model's attention mechanisms — respects the document's logical structure. A paragraph that states a claim and its supporting evidence stays together. A contract clause that defines an obligation and its exceptions stays together. A code function and its docstring stay together. When the AI retrieves a semantically coherent chunk, it can reason about the complete idea rather than reconstructing meaning from fragments.

The impact is most visible in question-answering scenarios. If a user asks 'what are the exceptions to the indemnification clause?' and the clause and its exceptions are in different chunks, the system may retrieve the clause without the exceptions (giving an incomplete answer) or the exceptions without the clause (giving a decontextualised answer). Semantic chunking that keeps the clause and its exceptions together enables a correct, complete response.

Fixed-size chunking splits text at regular token intervals, optionally with overlap between consecutive chunks. It is simple, deterministic, and works for any document type regardless of formatting. Its weakness is that it completely ignores meaning — a critical caveat may end up in a different chunk from the statement it qualifies, leading the AI to present the statement as unconditional when it actually has important limitations.

Paragraph or section-based chunking uses the document's existing structure (headings, paragraph breaks, list boundaries) as split points. This preserves logical units at the cost of producing chunks of varying size. Some chunks may be very short (a one-line heading) and others very long (a dense multi-paragraph section), which complicates retrieval scoring because chunk length affects embedding similarity calculations.

Semantic chunking uses embedding similarity to identify where the topic shifts within a document, splitting at points where adjacent sentences have low semantic similarity. This produces chunks that are topically coherent even in documents without clear structural markers like headings or paragraph breaks. The trade-off is computational cost — generating embeddings for every sentence in a large corpus requires significantly more processing than splitting on character count or structural markers.

Hierarchical chunking combines multiple levels: the full document is available as a summary, respecting token limits, sections are available as mid-level chunks, and paragraphs are available as fine-grained chunks. The retrieval system can match at the appropriate granularity — returning a full section when the query is broad and a specific paragraph when the query is narrow. This approach adds complexity to the retrieval pipeline but provides the most flexible and accurate results.

Chunk size involves a direct trade-off between context and precision. Larger chunks (1000+ tokens) provide more context per retrieval hit, which helps the AI understand the surrounding meaning and produce more nuanced answers. But larger chunks also dilute relevance — when a 1000-token chunk is retrieved because it contains one relevant sentence, the other 950 tokens are noise that competes for the model's attention and may confuse or distract from the relevant passage.

Smaller chunks (100-200 tokens) are more precise — each retrieved chunk is tightly relevant to the query. But they provide less context, which can lead to answers that are technically accurate but miss important qualifications, exceptions, or related points that were in adjacent chunks. An answer to 'is the service available 24/7?' might correctly identify 'yes' from a small chunk while missing the 'except during scheduled maintenance windows' that appears in the next chunk.

A common and effective approach is to retrieve small chunks but inject their surrounding context at query time — a form of context compression. When a 200-token chunk matches a query, the system retrieves it along with its parent section or adjacent chunks, giving the model both the precise match and the broader context. This combines the retrieval precision of small chunks with the comprehension benefits of larger context windows, though it requires the chunking pipeline to maintain parent-child relationships between chunks.

Adding overlap between consecutive chunks — where the end of one chunk repeats at the beginning of the next — mitigates the boundary problem in fixed-size chunking. If a relevant passage spans the boundary between two chunks, the overlap ensures that at least one chunk contains the complete passage. Common overlap values range from 10% to 25% of the chunk size.

The downside of overlap is redundancy. The same text appears in multiple chunks, which increases storage requirements, increases embedding computation costs, and can cause the retrieval system to return near-duplicate results. Deduplication at retrieval time — detecting when multiple retrieved chunks contain substantially the same content and merging or filtering them — is necessary to prevent the model from seeing the same passage multiple times.

For semantic or structural chunking methods, overlap is less necessary because the chunk boundaries already align with natural content boundaries. The text that would fall in the overlap zone of a fixed-size split is already contained within the correct semantic chunk. This is one of the practical advantages of investing in smarter chunking strategies: they reduce the need for overlap, which in turn reduces storage and retrieval complexity.

The most direct evaluation is end-to-end retrieval quality: given a set of test questions with known answers, does the retrieval system return chunks that contain the answer? If the correct chunk is not in the top results, the chunking strategy may be splitting the relevant information across multiple chunks, embedding it with too much surrounding noise, or creating chunks too small to match the query's semantic scope.

Chunk coherence can be evaluated independently of retrieval. Read a random sample of chunks and assess whether each one makes sense as a standalone unit. A chunk that begins mid-sentence, references an undefined term, or presents a conclusion without its premises is a sign that the chunking boundaries are cutting through logical units rather than between them. Coherent chunks produce better embeddings and better retrieval because their semantic content is self-contained.

Track retrieval metrics over time as your document corpus grows. A chunking strategy that works well for a hundred documents may degrade when the corpus reaches ten thousand, because the embedding space becomes more crowded and similar-seeming chunks from different documents compete for retrieval ranking. Regular evaluation ensures that chunking quality keeps pace with corpus growth and evolving query patterns.