What is Chunk Size?
Chunk Size is the token, character, or structural length chosen for each document unit indexed in a retrieval-augmented generation system.
How It Works
Chunk size controls how much source content each retrieval unit contains. Smaller units can localize evidence but may separate qualifiers, headers, or relationships; larger units can preserve context but may dilute ranking signals and consume more generation budget. No token count is a universal default. Production systems should derive a candidate grid from canonical evidence-span lengths, source structure, tokenizer limits, and query slices, then compare candidates under the same unique retrieved-token budget. The chosen size is a versioned policy tied to a corpus, tokenizer, embedding model, and evaluation set.
Key Characteristics
- Coordinate-aware: token length must be mapped back to canonical source offsets for citations and deletion
- Tokenizer-specific: the same text can have different lengths under different tokenizer versions
- Budget-coupled: size changes how many unique evidence tokens fit into retrieval and generation budgets
- Query-dependent: local facts, comparisons, procedures, and thematic questions can favor different granularities
- Experiment-selected: the policy is accepted only after retrieval, citation, answer, latency, and cost evaluation
Common Use Cases
- Building a candidate-size grid from the distribution of labeled evidence spans
- Testing whether legal clauses need parent expansion to include referenced definitions
- Keeping code symbols and their explanatory comments traceable to canonical offsets
- Comparing smaller and larger units under the same unique retrieved-token budget
- Gating a new size policy on retrieval, citation, answer, latency, and cost regressions
Example
Loading code...Frequently Asked Questions
What is a good default chunk size for RAG?
There is no transferable default. Derive candidate sizes from source structure, labeled evidence-span lengths, tokenizer limits, and query slices. Keep a simple fixed-token baseline, then select a policy only after equal-budget retrieval and end-to-end evaluation.
Are smaller chunks always better?
No. Smaller chunks can retrieve precise snippets, but they may omit definitions, assumptions, tables, or preceding context needed for a correct answer.
Why measure chunk size in tokens?
Embedding and generation APIs operate under token budgets, so token sizing helps control payloads and cost. Record the tokenizer and version, and retain canonical character or byte offsets because token coordinates alone are unstable across tokenizer changes.
Should every document use the same chunk size?
Not by assumption. A codebase, policy manual, and FAQ have different source structures and evidence needs. Use separate policies only when query-sliced evaluation justifies them, and version the policy so every indexed chunk remains reproducible.