What is Context Budget?
Context Budget is the planned allocation of a model's limited context-window tokens across instructions, user input, retrieved evidence, memory, tool data, and expected output.
How It Works
Context budget is the engineering discipline of deciding what deserves space in a prompt. Even models with large context windows have cost, latency, attention, and reliability limits. A good budget reserves space for system instructions, user input, retrieved evidence, conversation history, tool results, output format constraints, and the answer itself. It also defines what to summarize, compress, drop, or retrieve again when the request exceeds limits. Without a budget, prompts become expensive, noisy, and harder to debug.
Key Characteristics
- Allocates limited tokens across competing context needs
- Balances answer quality, cost, latency, grounding, and reliability
- Includes input context and reserved room for generated output
- Requires policies for truncation, summarization, retrieval, and compression
- Should be measured with the target model's tokenizer
Common Use Cases
- Reserving output space in long RAG prompts
- Limiting retrieved chunks to avoid noisy context stuffing
- Budgeting conversation history in chat assistants
- Separating system instructions from user documents and tool results
- Reducing TTFT by trimming unnecessary context
Example
Loading code...Frequently Asked Questions
Why does context budget matter with long-context models?
Large windows still cost money and latency, and models may underuse noisy or poorly placed information.
Should output tokens be part of the budget?
Yes. If the prompt consumes the full window, the model may not have enough room to produce the required answer.
How should RAG context be budgeted?
Prioritize high-quality evidence, preserve citations, avoid duplicates, and reserve space for instructions and answer generation.
What happens without a context budget?
Applications often stuff too much context, increasing cost, latency, missed evidence, and unpredictable behavior.