What is Context Caching?
Context Caching is a cross-request inference optimization that reuses provider- or runtime-managed state for an eligible prompt prefix, reducing repeated prefill computation without reusing the model's final answer.
Quick Facts
| Full Name | Context Caching / Prompt Caching / Prefix Caching |
|---|---|
| Specification | Official Specification |
How It Works
Context Caching, also called Prompt Caching or Prefix Caching by different systems, reuses work for input content that remains stable across requests. It is not a response cache: the model still generates a new completion from the cached prefix and the current suffix. It is also distinct from a per-request KV Cache, which accelerates token-by-token decoding, and from a Semantic Cache, which may return a stored answer for a similar query. Caching does not enlarge the Context Window or prove that the model used the cached evidence correctly.
The reusable unit is provider- and runtime-specific. Managed APIs may discover an eligible prefix implicitly, let the caller mark explicit breakpoints, or expose a separately created cached-content object. Self-hosted engines commonly hash full token blocks and reuse their computed attention state. In all cases, useful reuse depends on a compatible prefix identity, not merely on two prompts having similar meaning. Place stable instructions, Tool schemas, examples, and authorized reference material before timestamps, request IDs, retrieved passages that change per query, and the current user message.
A production cache identity must cover every value that can change the rendered prefix or computed state: model and Tokenizer revisions, chat template or serializer, ordered Tool and output schemas, multimodal preprocessing, adapters, runtime features, cache breakpoint, tenant or trust group, policy revision, and content hash. Provider APIs decide which fields participate in matching, so an application cache key is a routing or isolation hint, not proof that two requests are safe to share. Canonical serialization prevents accidental misses; authorization must still be enforced before content becomes cacheable.
Cache lifecycle is part of the contract. A cold request can compute and write a prefix, a warm request can read it, and an entry can expire or be evicted before its advertised maximum. Minimum cacheable length, Time to Live, refresh behavior, write charges, storage charges, breakpoint limits, and usage fields vary by model, endpoint, and provider. Concurrent requests may all miss if they arrive before the first write becomes visible. Cache unavailability must fall back to ordinary inference without changing correctness or authorization decisions.
Measure economics from observed traffic rather than advertised discounts. Record cache-read, cache-write, and uncached input tokens; hit and miss counts; reusable-prefix length; write amplification; evictions; p50 and p95 Time to First Token; end-to-end latency; and cost per successful task. Compare cold, warm, expired, changed-prefix, and caching-disabled controls. A high token hit rate can still be uneconomic when writes are rarely reused, storage is billed, routing fragments traffic, or the variable suffix dominates latency.
Treat cached state as sensitive derived data. Follow the provider's current retention, residency, and deletion contract; never assume that hashing makes secret content anonymous. Partition reuse by tenant and authorization scope, include a tenant salt or equivalent isolation input in self-hosted systems, and avoid cross-tenant sharing unless its confidentiality implications are explicitly accepted. Research has shown that cache-dependent timing can reveal whether another user's prefix was present and can enable prompt reconstruction in shared serving systems.
Release caching changes behind an observable, reversible policy. Pin the model, endpoint, prompt serializer, cache mode, key derivation, breakpoint layout, TTL, tenant scope, and pricing snapshot. Test exact hits, intended partial-prefix hits, misses, expiry, source updates, policy changes, concurrent cold starts, cancellation, failover, and cross-tenant denial. Alert when cache reads collapse, writes grow without later reuse, stale revisions remain addressable, or latency suggests sharing across forbidden scopes.
Key Characteristics
- Cross-request prefix reuse: avoids repeated prefill work while still generating a fresh response for each request
- Provider-specific semantics: implicit matching, explicit breakpoints, cached-content objects, thresholds, TTLs, and billing differ
- Strict cache identity: model, Tokenizer, serialization, ordered schemas, content, runtime features, and isolation scope can affect reuse
- Performance optimization, not correctness state: misses and evictions must degrade to uncached inference without changing policy
- Observable economics: cache reads, writes, misses, write amplification, TTFT, task cost, and expiry require separate metrics
- Security-sensitive shared state: tenant partitioning, retention, deletion, collision resistance, and timing side channels need controls
Common Use Cases
- Reusing a versioned system policy, Tool catalog, and output schema across many Agent turns
- Asking multiple questions about the same authorized document, code snapshot, audio file, or video
- Keeping append-only conversation prefixes warm while adding a new turn as the mutable suffix
- Sharing prefix blocks within one tenant or trust group in a self-hosted inference service
- Comparing implicit and explicit cache policies with cold, warm, expiry, and update traffic
Example
Loading code...Frequently Asked Questions
Are Context Caching, Prompt Caching, and Prefix Caching the same?
They often describe the same cross-request optimization: reuse processed state for a repeated input prefix. The names are not a universal API contract, however. Providers differ on implicit discovery, explicit breakpoints, cached-content objects, minimum length, matching, TTL, billing, and isolation, so implementations must follow the exact endpoint documentation.
How is Context Caching different from KV Cache and Semantic Cache?
A normal KV Cache reuses attention state during one autoregressive generation. Context or Prefix Caching extends compatible prefix reuse across requests. A Semantic Cache usually matches a similar query and may return a stored answer without running the model. These mechanisms have different keys, lifetimes, correctness risks, and security boundaries.
Does a cache hit change the answer or expand the Context Window?
It should not. A prefix hit skips compatible prefill work, but the model still generates a new response from the same logical input and current suffix. Caching does not add tokens to the model's Context Window, refresh stale source data, improve evidence use, or make a nondeterministic request return the same output.
Why is the cached-token count zero even when prompts look similar?
Common causes include a prefix below the model threshold, changed whitespace or ordering, a timestamp or user-specific field before the breakpoint, changed Tools or schemas, a different model or Tokenizer, an expired or evicted entry, fragmented routing, or concurrent requests arriving before the first cache write is available. Usage fields are the source of truth.
How should Context Caching be evaluated and secured?
Measure cold and warm TTFT, end-to-end latency, cache-read and write tokens, hit rate, write amplification, expiry, and task cost. Test changed prefixes, policy revisions, failover, and concurrency. Partition caches by tenant and authorization scope, review retention and residency, use salts where supported, and test timing-based cross-tenant disclosure.