What is Context Budget?
Context Budget is an application-level policy that allocates a model request's finite token capacity across serialized instructions, tools, user input, history, evidence, tool results, and generation reserve.
How It Works
A Context Budget is the application's allocation policy below a model endpoint's capacity limit. The Context Window states what the endpoint can accept; the budget states what this workload is allowed to spend and what must remain available for generation. A portable planning constraint is: serialized input plus generation reserve, provider or runtime allowance, and safety margin must not exceed the applicable model and endpoint limits. Some APIs expose one combined limit while others publish separate input, output, reasoning, or feature-specific caps, so no single subtraction formula applies to every provider.
Budget the assembled request, not only the visible user text. Common buckets include System and Developer policy, Tool schemas, current input, conversation history, retrieved evidence, memory, examples, Tool Call and Result groups, multimodal representations, and output-format schemas. Give each bucket a hard cap, a soft target, a protection class, and an overflow action. Fixed percentages are only hypotheses: a document-QA route may spend most of its input on evidence, while an Agent may need more Tool and state capacity. Unused capacity may be borrowed only through an explicit policy that preserves mandatory floors and the total ceiling.
Count after the final Chat Template or provider serialization with the target model's Tokenizer or token-count endpoint. Role wrappers, JSON Schema, Tool descriptions, images, PDFs, audio, and provider-added formatting can make a raw-text estimate wrong. Token-count endpoints may themselves be estimates, so keep measured headroom and reconcile planned counts with response Usage. Bind the policy to model, endpoint, Tokenizer, serializer, Prompt, Tool catalog, retrieval, and output-schema revisions; recalculate it when any identity changes.
Overflow behavior is part of the contract. Protected System and safety policy, authorization scope, the current task, required output Schema, source identifiers, unresolved conflicts, and atomic Tool Call/Result groups must not be silently cut. Elastic history can use a sliding window or verified checkpoint; retrieved evidence can be authorized, reranked, deduplicated, narrowed, or split; Tool results can be projected, paginated, or stored behind a scoped reference. If required content still cannot fit, fail explicitly, ask the user to narrow the task, route to a compatible endpoint, or split the work instead of relying on undocumented truncation.
A per-call Context Budget is not an Agent run budget, billing cap, rate limit, or permission system. A multi-step Agent also needs limits for model calls, Tool calls, wall time, retries, bytes, and total tokens across the run. Every Tool result can become the next request's input, so validate type and size before inclusion and preserve call/result atomicity. Authorize and classify data before budgeting it: removing content to fit does not make unauthorized data safe, and an oversized untrusted result can be a cost or availability attack.
Prompt Caching can reduce repeated prefill work or price, but cached tokens still occupy logical context and do not increase the budget. Context Compression can reduce a bucket but introduces a Loss Budget and provenance obligations. Record planned and actual tokens by bucket, overflow reason and action, omitted or transformed source IDs, generation usage, provider-reported reasoning or cached tokens, finish reason, latency, cost, and task outcome without logging secrets or hidden reasoning text.
Evaluate the policy as part of the complete release. Compare several budget variants against an unbudgeted or larger-context control using the same model, data, and output contract. Measure task success, critical-evidence and policy-constraint recall, citation support, incomplete-output and overflow rates, p50/p95/p99 utilization by bucket, Time to First Token, end-to-end latency, and cost per successful task. Include multilingual text, code, large schemas, multimodal inputs, long sessions, oversized Tool results, conflicting evidence, Prompt Injection, provider revision changes, and cases where no safe allocation exists.
Key Characteristics
- Application policy, not model capacity: the Context Window is a ceiling, while the budget defines workload-specific allocation below it
- Complete request accounting: serialized instructions, Tools, history, evidence, media, generation, runtime allowance, and headroom are explicit
- Bucket contracts: every category has a cap, target, protection class, priority, and deterministic overflow action
- Revision-bound measurement: model, endpoint, Tokenizer, serializer, Prompt, Tool catalog, retrieval, and Schema identity affect the count
- Safe overflow semantics: protected content is retained or the request is narrowed, split, rerouted, reviewed, or rejected
- Outcome-based validation: utilization and token savings are evaluated with quality, evidence preservation, latency, cost, and abuse tests
Common Use Cases
- Assembling RAG requests with authorized evidence, citation metadata, and a protected generation reserve
- Bounding long conversations with raw recent turns, verified checkpoints, and retrievable source history
- Loading only task-relevant Tool schemas and projecting or paginating oversized Tool results
- Routing multilingual, code, tabular, or multimodal requests using model-specific token accounting
- Enforcing per-call capacity inside a separately bounded multi-step Agent execution
Example
Loading code...Frequently Asked Questions
How is a Context Budget different from a Context Window?
The Context Window is a model-and-endpoint capacity limit. A Context Budget is the application's policy for spending less than that limit across instructions, Tools, current input, history, evidence, results, and generation reserve. A request can fit the window yet violate its budget because it is too costly, slow, noisy, or risky for the workload.
How should an application calculate a Context Budget?
Read the exact model and endpoint limits, reserve generation capacity and measured safety headroom, then count the complete serialized input with the target Tokenizer or provider endpoint. Include role wrappers, Tool and output schemas, history, retrieved evidence, Tool results, and multimodal representations. Reconcile the estimate with actual Usage and version the policy.
Do cached tokens still consume the Context Budget?
Yes. Prompt or Context Caching can reduce repeated prefill computation or price, but the cached prefix remains part of the logical input and still occupies the Context Window. Caching does not create extra capacity, authorize stale content, or remove the need to reserve output and test effective context use.
What should happen when a request exceeds its Context Budget?
Apply the declared action for the overflowing bucket: load fewer Tool schemas, narrow and rerank evidence, compact history, project or paginate Tool results, split the task, or route to a compatible endpoint. Never silently remove protected policy, authorization, the current task, required Schema, source identity, or half of a Tool Call/Result group.
Should every LLM application use the same budget percentages?
No. Allocation depends on the task, model, language, Tool catalog, retrieval distribution, output contract, risk, latency target, and cost objective. Treat percentages as candidates, then compare budget variants on representative and adversarial evaluations. Keep the smallest allocation that passes quality, safety, and operational gates.