Core Summary
LLM semantic caching is approximate reuse of a complete prior response. It can remove an expensive generation call, but a false hit can silently answer a different question, reveal another tenant's data, or replay stale policy. A safe hit therefore has two gates: exact equality for every hard scope field, followed by a calibrated semantic comparison for the request representation. The cache must also support admission, invalidation, bypass, and evidence-based rollout.
Distinguish the Caches
| Mechanism | Reused object | Match | Does generation run? | Main failure |
|---|---|---|---|---|
| exact response cache | complete answer | canonical request hash | no | incomplete key or stale entry |
| semantic response cache | complete answer | hard filters plus vector similarity | no | plausible answer for a different intent |
| provider prompt cache | prefix computation | provider-defined token prefix | yes | assuming portable hit and billing semantics |
| KV cache | attention state | token and runtime identity | yes | capacity, isolation, and invalid reuse |
| RAG retrieval | evidence passages | relevance query | yes | missing, stale, or unauthorized evidence |
A semantic cache may use a vector database and embeddings, but it is not RAG. RAG retrieves evidence for a new answer; semantic response caching may skip the model and replay an old answer.
Define the Cacheable Unit
Start with one route and one response type. Good candidates are public, stable, repetitive, read-only questions with a canonical answer. Poor candidates include:
- personalized account or health information;
- authorization, fraud, legal, medical, or financial decisions;
- inventory, price, incident status, and other rapidly changing facts;
- open-ended creation where wording is the product;
- Tool calls, writes, purchases, deletes, or other side effects;
- conversations where omitted history changes the meaning.
Do not cache a request merely because it received HTTP 200. A refusal, truncated stream, unsupported claim, or answer generated during a downstream outage is usually not reusable.
Build a Complete Cache Identity
The embedding is only one field. Partition or filter on every non-semantic input that can change the valid answer:
{
"tenant": "tenant-a",
"subject_class": "member",
"authorization_scope": "support:public",
"task": "returns-policy",
"locale": "en-US",
"conversation_revision": "standalone-v2",
"model_revision": "model-family/rev-8",
"prompt_revision": "support-answer/v12",
"toolset_revision": "read-only-tools/v4",
"source_revision": "kb/rev-31",
"policy_revision": "safety/v8",
"output_schema": "answer/v3",
"freshness_class": "24h"
}
Apply tenant and authorization filters inside or before nearest-neighbor search. Searching a global index and discarding the top result afterward can leak existence through timing, ranking, metrics, or diagnostics, and it can miss the best permitted candidate.
Use a task-specific canonical request representation. The last user message may omit relevant conversation state; the full transcript may include personal data and irrelevant history. Define which fields affect meaning, version that representation, and test it.
A Two-Gate Lookup
The following runnable Go example first checks an exact partition and freshness, then applies cosine similarity:
package main
import (
"fmt"
"math"
"time"
)
type Scope struct {
Tenant string
Authorization string
Locale string
PromptRevision string
SourceRevision string
PolicyRevision string
}
type Entry struct {
Scope Scope
Vector []float64
Response string
ExpiresAt time.Time
}
func cosine(left, right []float64) float64 {
if len(left) == 0 || len(left) != len(right) {
return -1
}
var dot, leftNorm, rightNorm float64
for index := range left {
dot += left[index] * right[index]
leftNorm += left[index] * left[index]
rightNorm += right[index] * right[index]
}
if leftNorm == 0 || rightNorm == 0 {
return -1
}
return dot / (math.Sqrt(leftNorm) * math.Sqrt(rightNorm))
}
func lookup(entry Entry, scope Scope, vector []float64, threshold float64, now time.Time) (string, bool) {
if entry.Scope != scope || !now.Before(entry.ExpiresAt) {
return "", false
}
if cosine(entry.Vector, vector) < threshold {
return "", false
}
return entry.Response, true
}
func main() {
now := time.Date(2026, time.October, 4, 10, 0, 0, 0, time.UTC)
scope := Scope{
Tenant: "public",
Authorization: "docs:read",
Locale: "en-US",
PromptRevision: "v12",
SourceRevision: "kb-31",
PolicyRevision: "v8",
}
entry := Entry{
Scope: scope,
Vector: []float64{1, 0.2, 0},
Response: "Returns are accepted within the published policy window.",
ExpiresAt: now.Add(time.Hour),
}
answer, hit := lookup(entry, scope, []float64{0.98, 0.21, 0.01}, 0.99, now)
fmt.Printf("hit=%t answer=%q\n", hit, answer)
otherTenant := scope
otherTenant.Tenant = "tenant-b"
_, hit = lookup(entry, otherTenant, []float64{0.98, 0.21, 0.01}, 0.99, now)
fmt.Printf("cross_tenant_hit=%t\n", hit)
}
Expected output:
hit=true answer="Returns are accepted within the published policy window."
cross_tenant_hit=false
The numeric threshold is only a fixture value. In a real vector store, confirm whether the API returns similarity or distance, the range and ordering of that score, normalization behavior, approximate-search recall, and filter semantics.
Calibrate Thresholds on Harm
There is no universal 0.8, 0.95, or other safe threshold. The same number means different things across embedding models and distance metrics.
Create labeled pairs from real, redacted traffic:
| Pair | Expected |
|---|---|
| "How do I return an item?" / "What is the return process?" | reusable |
| "Can final-sale items be returned?" / "How do I return an item?" | usually not reusable |
| "Reset my password" / "Reset another user's password" | not reusable |
| same wording, different tenant | blocked before similarity |
For each task and risk slice, sweep thresholds and measure:
- eligible-request rate;
- useful-hit rate;
- false-hit and harmful-false-hit rate;
- stale-hit rate;
- abstention and fallback quality;
- p50, p95, and p99 lookup and end-to-end latency;
- embedding, storage, invalidation, and generation cost;
- cost per accepted answer.
The 2024 GPT Semantic Cache preprint reports large API-call reductions in its own generated-query experiment and selected 0.8 for that setup. It also identifies limited query complexity, synthetic test generation, threshold sensitivity, and simplified evaluation as limitations. Use it as evidence that the technique can work, not as a production default.
Control Admission
Write an entry only when:
- the route and response type are cache-eligible;
- generation completed and the output schema is valid;
- required evidence and citations passed;
- no secret or personalized field remains;
- source, model, prompt, policy, and evaluator revisions are known;
- expiry and invalidation triggers are defined;
- the answer passed deterministic and, where necessary, human review.
Record the source answer's evidence IDs and validation state. A cache should amplify only responses that already satisfy the direct-response contract.
Invalidate by Event and Version
TTL limits age but does not express why an answer became invalid. Rotate a namespace or invalidate entries when:
- a source changes, is corrected, expires, or is deleted;
- access or tenant ownership changes;
- the model, embedding, prompt, Tool set, policy, or schema changes;
- a safety incident affects an answer class;
- a user reports a harmful response;
- the canonical representation changes.
Prefer immutable versioned namespaces plus expiry over scanning and mutating millions of entries. Maintain a reverse index from source and policy revisions to cache entries when urgent invalidation is required.
Sliding TTL can keep a popular stale answer alive indefinitely. Use absolute expiry for policy and knowledge answers unless a successful revalidation refreshes the source revision.
Prevent Stampedes and Partial Entries
Concurrent misses for the same canonical identity can trigger duplicate model calls. Use bounded request collapsing:
- elect one short-lived leader for an exact cache identity;
- let followers wait only within their own deadlines;
- propagate cancellation correctly;
- never hold a global lock across unrelated tenants;
- do not admit partial streams after timeout or cancellation;
- return a fresh fallback if the cache is unavailable.
Negative caching is appropriate only for deterministic, short-lived outcomes such as "resource not found" under an exact authorization scope. Do not semantically reuse denials, safety refusals, or rate-limit errors across users.
Defend Against Cache Poisoning
An attacker may craft a query or retrieved source that produces a plausible malicious answer, then rely on the cache to replay it. Controls include:
- admission only after evidence and policy validation;
- trusted writers and authenticated invalidation;
- per-route and per-tenant partitions;
- bounded text, metadata, vectors, and response size;
- source provenance and content digests;
- anomaly detection for sudden hit concentration;
- sampled bypass to compare with fresh generation;
- immediate kill switch and purge by source or policy revision.
Encryption at rest does not make an incorrectly shared entry safe.
Roll Out in Stages
- Offline replay: label pairs and estimate the safe operating region.
- Shadow lookup: compute candidates without serving them.
- Reviewed public intents: enable only stable, non-personalized routes.
- Limited canary: expose a small traffic slice with instant bypass.
- Sampled fresh generation: audit whether hits remain equivalent.
- Controlled expansion: add slices only when harmful false hits remain within policy.
Monitor lookup errors, filter misses, false hits, stale hits, hit age, invalidation lag, bypasses, tail latency, and cost. Raw prompts are rarely required in operational logs; use redacted intent and digests where possible.
Frequently Asked Questions
What is semantic caching for LLMs?
It is approximate reuse of a complete response. The system first matches hard scope and version fields, then applies a calibrated semantic rule to the request representation.
How is it different from prompt caching?
Semantic response caching can skip generation. Provider prompt caching reuses eligible prefix computation but still produces a new continuation.
What belongs in the key?
Every field that changes answer validity: tenant, authorization, task, locale, conversation representation, model, prompt, Tool set, source, policy, schema, and freshness.
How do I choose a threshold?
Use labeled positive and negative pairs from the target workload. Optimize for harmful false-hit tolerance and accepted-answer value by slice, not a copied number.
Can a semantic cache be shared across users?
Only for genuinely public and invariant answers. Personalized or permission-sensitive entries must remain in an exact trusted partition, and many should not be cached at all.