TL;DR

An LLM semantic cache reuses complete answers for meaningfully similar requests. Similarity alone is unsafe. A production hit requires a calibrated threshold and matching tenant, authorization scope, locale, model, prompt, source revision, safety policy, and freshness window. Measure false hits by intent slice, keep entries non-authoritative, and make invalidation and bypass observable.

Semantic cache, exact cache, prompt cache, and KV cache

The word “cache” covers several different mechanisms.

Cache type Reused object Match key Does generation run?
Exact response cache Complete answer Exact normalized request No
Semantic response cache Complete answer Embedding similarity + filters No
Provider prompt cache Model prefix computation Repeated token prefix Yes
KV cache Attention keys and values Tokens within or across supported requests Yes
RAG retrieval Source passages Query relevance Yes

A semantic cache is closest to an approximate response cache. It is not a vector database used for RAG, even though both may use vector search and an embedding. RAG retrieves evidence for a new answer; a semantic cache returns a prior answer.

The production hit path

flowchart LR A["Request"] --> B["Normalize and classify"] B --> C["Apply tenant and policy filters"] C --> D["Embed request"] D --> E["Nearest cached candidate"] E --> F{"Distance and freshness pass?"} F -->|yes| G["Return validated cached answer"] F -->|no| H["Run retrieval and LLM"] H --> I["Validate response"] I --> J["Write versioned cache entry"]

A valid lookup should include:

json 
{
  "tenant": "tenant_42",
  "auth_scope": "support:public",
  "locale": "en-US",
  "model_version": "model-family/revision",
  "prompt_version": "support-answer/v12",
  "source_revision": "kb-2026-07-28",
  "safety_policy": "policy-v8",
  "response_type": "public_faq"
}

Perform these filters inside the vector query where possible. Fetching a global nearest neighbor and checking tenant afterward creates avoidable leakage and timing risks.

Thresholds are a risk decision

A loose threshold increases hit rate and false matches. A strict threshold reduces false hits but can make the cache economically irrelevant. The correct threshold depends on the embedding model, distance metric, normalization, request distribution, and harm of returning a wrong answer.

Build a labeled pair set:

Pair Expected
“How do I return an item?” / “What is the return process?” match
“Can I return final-sale items?” / “How do I return an item?” often no match
“Reset my password” / “Reset another user's password” no match
same question across tenants no cross-partition match

Sweep thresholds and report:

  • eligible request rate;
  • hit rate;
  • false-hit rate;
  • harmful false-hit rate;
  • stale-hit rate;
  • p50/p95 lookup and total latency;
  • cost per successful answer.

Do this by intent and risk slice. An aggregate 1% false-hit rate can be unacceptable if it concentrates in account, medical, or payment questions.

Key design and invalidation

The cache key is more than an embedding. Version every input that changes answer meaning.

typescript 
type CachePartition = {
  tenant: string;
  authScope: string;
  locale: string;
  modelVersion: string;
  promptVersion: string;
  sourceRevision: string;
  policyVersion: string;
};

function canReuse(
  candidate: CachePartition,
  request: CachePartition,
): boolean {
  return Object.keys(request).every(
    (key) =>
      candidate[key as keyof CachePartition] ===
      request[key as keyof CachePartition],
  );
}

Use TTL for time-based freshness, but do not rely on TTL alone. Trigger invalidation or version rotation when:

  • a source document changes or is deleted;
  • permissions or tenant ownership change;
  • the prompt, model, or embedding model changes;
  • safety policy changes;
  • a response is corrected or reported harmful;
  • an incident invalidates a class of answers.

Prefer versioned namespaces over scanning and mutating every entry. Old entries can expire naturally while new requests use the current namespace.

What belongs in an entry

Store enough evidence to decide whether the entry is reusable, but minimize sensitive content.

json 
{
  "request_embedding": "<vector>",
  "normalized_request_hash": "sha256:...",
  "response": "<bounded validated answer>",
  "source_ids": ["policy-v7#returns"],
  "partition": { "tenant": "public", "locale": "en-US" },
  "versions": { "prompt": "v12", "model": "rev-8", "policy": "v8" },
  "created_at": "2026-07-28T10:00:00Z",
  "ttl_seconds": 86400,
  "validation": { "status": "passed", "suite": "faq-v4" }
}

Do not cache secrets, raw credentials, one-time tokens, hidden authorization decisions, or private model context. Encryption at rest does not make cross-tenant reuse safe.

Cache admission policy

Not every successful response should enter the cache. Admit only responses that pass an explicit policy:

  • request classified as cache-eligible;
  • no personalized or secret fields;
  • deterministic policy and safety checks passed;
  • answer has required citations or evidence;
  • source revision and expiry are known;
  • output is complete, not refused or truncated;
  • response type is safe to replay.

An admission policy prevents one low-quality or attacked response from becoming a repeated failure.

Failure modes

Failure Impact Control
False semantic hit plausible answer to wrong intent labeled threshold evaluation and intent routing
Cross-tenant hit data disclosure pre-filtered partitions and trusted identity
Stale hit obsolete policy or facts source versions, TTL, event invalidation
Cache poisoning malicious answer repeatedly served admission validation and provenance
Model/prompt drift behavior inconsistent with current release versioned namespaces
Popularity lock-in old answer suppresses improved generation sampling bypass and periodic refresh
Cache outage latency spike or request failure cache-as-optional dependency and bounded timeout

The cache should be safe to bypass and rebuild. It must not become the sole system of record.

Evaluation and rollout

Roll out in stages:

  1. Shadow mode: run lookups but never serve hits; collect candidate pairs.
  2. Read-only canary: serve only reviewed public intents and strict thresholds.
  3. Limited production: enable selected tenants or routes with instant kill switch.
  4. Threshold tuning: expand only after reviewing false-hit slices.
  5. Ongoing audit: bypass a sample of hits to compare fresh generation and detect drift.

For each hit, record entry ID, partition digest, distance, threshold, age, source revision, validation version, bypass reason, and outcome. Avoid logging raw prompts when a digest and classified intent are sufficient.

When semantic caching is a good fit

Use it for high-volume, stable, repeatable, low-personalization questions where a complete response can be safely reused. FAQ assistants, public documentation, and bounded classification are common candidates.

Avoid or narrowly constrain it for:

  • account-specific answers;
  • authorization and security decisions;
  • rapidly changing inventory, price, or status;
  • legal, financial, or medical conclusions;
  • writes and side effects;
  • requests where conversational context changes meaning.

The LLM gateway architecture guide explains where cache policy fits among routing and budgets. The AI inference cost guide covers how to measure savings without hiding quality regressions.

FAQ

Should semantic similarity be calculated on the full conversation?

Usually not without deliberate summarization and versioning. Conversation history can contain identity, preferences, prior decisions, and instructions that change meaning. A last-message-only key can also be wrong. Define a task-specific canonical representation.

Can one cache serve multiple embedding models?

Do not mix vectors from incompatible embedding spaces in one index. Version the embedding model and rebuild or dual-read during migration.

Should a cache hit refresh TTL?

It depends on freshness semantics. Sliding TTL can keep a popular obsolete answer alive indefinitely. Absolute expiry tied to source revision is safer for policy and knowledge content.

How do streaming responses interact with caching?

Cache only completed, validated responses. Never admit a partial stream after cancellation or timeout. Streaming and non-streaming response formats may need separate entries.

How do I prevent cache poisoning?

Apply the same evidence, policy, safety, and completion checks used for direct responses before admission. Record provenance and support immediate invalidation by entry, source, version, and policy.

Sources