What is Chunking?
Chunking is the process of splitting long documents or data sources into smaller retrievable units that preserve enough semantic context for embedding, indexing, retrieval, and grounded generation.
How It Works
Chunking is one of the highest-leverage design choices in RAG because it defines the units that can be indexed, retrieved, cited, authorized, updated, and deleted. A chunk may be a paragraph, heading section, table region, code symbol, or token window, but it should remain a traceable view over canonical source content. Structure-aware splitting follows author or parser boundaries; semantic chunking infers boundaries from embeddings or model signals. Neither is universally superior. Production systems should preserve source offsets and lineage, version every transformation, and compare candidates on evidence-labeled queries under equal retrieval budgets.
Key Characteristics
- Retrieval unit design: defines the smallest indexed unit a retriever can return
- Canonical lineage: each unit retains source URI, version, offsets, content hash, and access scope
- Boundary policy: may use author structure, fixed token windows, inferred semantic changes, or parent-child relationships
- Independent retrieval and context units: a precise child hit can expand to a separately authorized parent or region
- Versioned transformation: parser, tokenizer, splitter, embedding model, and derived-context versions remain reproducible
- Evaluation-dependent: candidates are compared on evidence recall, ranking, duplication, citations, latency, and update cost
Common Use Cases
- Splitting product documentation into heading-aware sections for RAG
- Creating paragraph-level chunks for policy question answering
- Preserving code blocks and surrounding explanation in developer documentation
- Separating tables or forms into retrievable records with metadata
- Testing fixed-token, structure-aware, semantic, and parent-child candidates under the same unique-token budget
Example
Loading code...Frequently Asked Questions
Why is chunking important in RAG?
Chunking defines what the retriever can return. If chunks are poorly shaped, the system may retrieve incomplete evidence, mix unrelated topics, lose citation boundaries, or waste context-window budget.
Is fixed-size chunking enough?
It is a necessary baseline, not a universal winner or loser. Structure-aware splitting is another candidate when headings, tables, lists, code symbols, or policy sections carry useful boundaries, but parser quality and query distribution can reverse the result. Compare both under the same unique retrieved-token budget.
How should chunking be evaluated?
Label canonical evidence spans and compare candidates under the same unique retrieved-token budget. Measure evidence recall, ranking quality, duplicate-context ratio, citation correctness, answer faithfulness, latency, index size, update cost, authorization leakage, and deletion propagation.
Can chunking leak restricted data?
Yes. Never merge content across access boundaries, and do not treat metadata as enforcement. Reauthorize the source object when retrieving a chunk or expanding a child to a parent, then propagate revocation and deletion through chunks, summaries, embeddings, and caches.