KV Cache for LLM Inference: Memory, Reuse, and Tuning

KV cache is the per-layer attention state retained for tokens a causal language model may need again. It removes repeated Key and Value projections during autoregressive decoding, but it also consumes memory, bandwidth, and scheduler capacity. A production design must therefore answer four questions: which token states remain resident, when they may be reused, where they live, and how their correctness and isolation are verified.

This guide owns that cache lifecycle. The LLM inference guide covers the full Prefill/Decode pipeline, while the disaggregated serving guide covers moving KV state between worker pools.

Direct Answer

KV cache exchanges repeated projection work for persistent state. Prefill creates K/V tensors for prompt tokens; every Decode step reads retained K/V, computes Q/K/V for the newest token, and appends the new K/V state.

How KV Cache Works

In causal self-attention, an earlier token cannot depend on a future token. Once the model has computed an earlier position's Key and Value tensors for a fixed model and input history, later Decode steps can reuse those tensors.

For a new token at position t, each attention layer:

1. computes the new token's Query, Key, and Value;
2. compares the new Query with retained Keys;
3. combines retained Values using the resulting attention weights;
4. appends the new Key and Value to that layer's cache.

The logical operation is:

K_cache <- concat(K_cache, k_t)
V_cache <- concat(V_cache, v_t)

output_t = softmax(q_t @ K_cache.T / sqrt(head_dim)) @ V_cache

The cache keeps K and V because future Queries need them. A Query is consumed by its current position and is not reused by later positions. Attention scores are also not reusable because every new Query produces a different score vector.

This explanation is scoped to causal Transformer attention. Encoder-decoder models may maintain separate self-attention and cross-attention caches. Sliding-window, chunked, latent-attention, recurrent, and hybrid architectures can retain different state and require model-specific formulas.

Prefill and Decode Have Different Cache Behavior

Prefill creates the initial cache in parallel, while Decode grows or updates it one step at a time. That distinction determines which optimization can improve which latency metric.

The Hugging Face Caching documentation describes the per-layer append behavior and warns that caching belongs to inference, not training. Training changes weights and processes sequences under different execution semantics, so an inference cache is not a training shortcut.

Calculate KV Cache Memory from Resident Tokens

For a basic uncompressed decoder-only cache, start with bytes per retained token:

bytes_per_token =
    2
    * n_layers
    * n_kv_heads
    * head_dim
    * bytes_per_element

logical_cache_bytes =
    bytes_per_token
    * resident_token_slots

The factor 2 represents Key and Value. n_kv_heads is the number of KV heads, not necessarily the number of Query heads. For active sequences with different retained lengths:

resident_token_slots = sum(retained_tokens_for_each_active_sequence)

This logical estimate is not the physical GPU allocation. Add or measure:

• fixed-size reservation or page rounding;
• block tables, scales, metadata, and allocator overhead;
• duplicated versus physically shared prefixes;
• speculative or beam-search branches;
• tensor/pipeline parallel placement and replication;
• temporary attention workspaces and activations;
• model-specific cross-attention, sliding-window, or latent state.

A model's advertised context window is only an upper bound. It becomes a physical cache reservation when the chosen layout actually preallocates that bound. Dynamic and paged engines may allocate closer to retained tokens; static caches may reserve more; sliding-window layers stop retaining tokens beyond their trained window.

Run a Capacity Estimate

The following dependency-free calculator estimates logical cache memory from active sequence lengths. It deliberately excludes quantization metadata and page overhead so those costs remain visible rather than hidden in a false precision.

from __future__ import annotations

from dataclasses import dataclass
from typing import Iterable


GIB = 1024 ** 3


@dataclass(frozen=True)
class KVCacheSpec:
    n_layers: int
    n_kv_heads: int
    head_dim: int
    bytes_per_element: int

    def bytes_per_token(self) -> int:
        values = (
            self.n_layers,
            self.n_kv_heads,
            self.head_dim,
            self.bytes_per_element,
        )
        if any(value <= 0 for value in values):
            raise ValueError("KV cache dimensions must be positive")
        return (
            2
            * self.n_layers
            * self.n_kv_heads
            * self.head_dim
            * self.bytes_per_element
        )


def logical_cache_gib(
    spec: KVCacheSpec,
    retained_sequence_lengths: Iterable[int],
) -> float:
    lengths = tuple(retained_sequence_lengths)
    if not lengths or any(length < 0 for length in lengths):
        raise ValueError("Provide one or more non-negative sequence lengths")
    resident_tokens = sum(lengths)
    return spec.bytes_per_token() * resident_tokens / GIB


active_sequences = [4096, 2048]
mha = KVCacheSpec(
    n_layers=32,
    n_kv_heads=32,
    head_dim=128,
    bytes_per_element=2,
)
gqa = KVCacheSpec(
    n_layers=32,
    n_kv_heads=8,
    head_dim=128,
    bytes_per_element=2,
)

try:
    print(f"MHA logical cache: {logical_cache_gib(mha, active_sequences):.3f} GiB")
    print(f"GQA logical cache: {logical_cache_gib(gqa, active_sequences):.3f} GiB")
except ValueError as error:
    raise SystemExit(f"Invalid capacity input: {error}") from error

# MHA logical cache: 3.000 GiB
# GQA logical cache: 0.750 GiB

The fourfold difference comes only from changing 32 KV heads to 8 under the stated dimensions. It is not a promise that an arbitrary MHA checkpoint can switch to GQA at runtime.

Treat Cache Identity as a Correctness Contract

A reusable block is valid only for the exact computation that produced it. Token equality is necessary, but production identity often includes more than visible text.

{
  "modelId": "provider/model-name",
  "modelRevision": "immutable-revision",
  "tokenizerRevision": "immutable-revision",
  "adapterId": "none",
  "chatTemplateVersion": "chat-v3",
  "attentionConfigHash": "sha256:...",
  "kvDtype": "bf16",
  "multimodalInputHashes": [],
  "trustScope": "tenant-acme",
  "cacheSaltId": "salt-ref-42"
}

The identity should cover every input capable of changing K/V state:

• model weights and immutable revision;
• tokenizer, special tokens, and chat template;
• LoRA or other adapter identity;
• positional/attention configuration and cache dtype;
• multimodal embeddings or input hashes;
• engine compatibility where layouts are not portable;
• tenant or trust-group isolation policy.

vLLM's prefix caching design hashes block tokens, parent-prefix identity, and extra values such as LoRA IDs and multimodal hashes. Its V1 design also supports a per-request cache salt to isolate trust groups and reduce timing-based disclosure. That is a vLLM feature, not a substitute for application authorization or data-retention policy.

Compare Cache Strategies by Their Actual Trade-off

Cache strategies operate on different axes and can be combined. "Use KV cache" is not enough to choose a layout.

Hugging Face's current Cache strategies distinguish DynamicCache, StaticCache, QuantizedCache, and offloaded variants. The documentation explicitly frames static cache as a compilation-versus-waste trade-off, offload as memory-versus-transfer work, and quantization as potentially slower for short contexts when memory is already sufficient.

What PagedAttention Does and Does Not Prove

PagedAttention changes physical KV memory management, not the model's attention semantics. It partitions a sequence's logical cache into blocks that can be allocated on demand and mapped to non-contiguous physical memory.

This supports a lifecycle similar to a memory manager:

1. reserve block capacity for an admitted request;
2. allocate blocks as Prefill and Decode create token state;
3. share read-only prefix blocks when identity matches;
4. copy or allocate blocks when branches diverge;
5. free or mark blocks reusable when requests finish;
6. evict unreferenced reusable blocks under pressure.

The original PagedAttention paper reported 2-4x higher throughput at similar latency than FasterTransformer and Orca in its evaluated models and workloads. That result is evidence for the design, not a universal multiplier for current engines, hardware, sequence distributions, or decoding policies.

Paging also does not eliminate all waste. The final block can be partially filled; metadata and indirection remain; cache kernels and block size affect efficiency; and a scheduler can still overcommit token capacity.

Prefix Caching Optimizes Prefill, Not Decode

Prefix caching reuses compatible K/V state for a shared token prefix across requests. Its benefit depends on the number of reused tokens, cache hit rate, lookup and transfer cost, and how long the blocks survive before eviction.

vLLM's Automatic Prefix Caching guide names repeated long-document queries and multi-round conversations as useful workloads. It also states the central limit: prefix caching reduces query processing in Prefill but does not reduce the time to generate new output tokens during Decode.

Measure at least:

reused_token_ratio = reused_prefix_tokens / eligible_prefix_tokens
hit_rate           = requests_with_reuse / eligible_requests

A raw request hit rate can mislead. Reusing 16 tokens and reusing 16,000 tokens both count as one hit, but they do not save the same Prefill work.

Provider "prompt caching" should be treated as a product contract. It may expose token thresholds, TTLs, discounts, or automatic matching without promising a particular KV implementation. Do not infer an engine layout, cross-request retention period, or security boundary from the feature name.

Quantization and Offload Need Separate Benchmarks

KV quantization and cache offload both trade one bottleneck for another. Neither is an automatic low-memory mode.

The KIVI paper observed different Key and Value distributions and proposed asymmetric 2-bit quantization: per-channel for Keys and per-token for Values. Its memory, batch, throughput, and quality results belong to the evaluated models and implementation. Other dtypes, kernels, residual-cache lengths, and models require new measurements.

For quantization, compare:

• logical and physical cache bytes;
• quantize/dequantize kernel time;
• TTFT, TPOT, and throughput by context length;
• task quality and deterministic regression slices;
• scale, residual-cache, padding, and metadata overhead.

For offload, compare:

• bytes transferred per request and per generated token;
• host-to-device and device-to-host queue time;
• cache hit latency by memory tier;
• link utilization under concurrent Prefill and Decode;
• recompute cost when transfer is slower than rebuilding.

TensorRT-LLM's KV cache reuse documentation describes host offload as a way to retain reusable blocks longer, while warning that transfers have a cost and older interconnects may not benefit. That conclusion is hardware- and workload-dependent.

GQA and MQA Are Model Architecture Choices

Grouped-Query Attention and Multi-Query Attention reduce n_kv_heads, which lowers cache bytes and bandwidth at the source. They are normally properties of a trained checkpoint, not serving flags that can be enabled without changing the model.

• MHA uses a KV head for each Query head.
• MQA shares one KV head across Query heads.
• GQA uses an intermediate number of KV heads shared by Query groups.

The MQA paper targets incremental Decode bandwidth by sharing Keys and Values. The GQA paper presents GQA as a quality/efficiency intermediate and evaluates an uptraining method. Use a checkpoint's actual configuration in capacity calculations; do not infer head counts from parameter size or model family name.

Capacity Planning Must Use a Workload Distribution

Admission control should reserve for resident token demand, not a headline request count. Ten short requests and ten maximum-context requests are not equivalent, and continuous batching changes that active set at each scheduling iteration.

A useful planning workflow is:

1. collect prompt, output, retained-context, and concurrency distributions;
2. calculate bytes per token from the exact checkpoint and cache dtype;
3. model page rounding, shared prefixes, branches, and safety headroom;
4. subtract weights, activations, workspaces, and non-cache runtime memory;
5. derive token-capacity admission limits;
6. replay the distribution under realistic arrival bursts;
7. verify OOM, preemption, and latency behavior at the SLO boundary.

Bound both input and output tokens. An output limit only bounds future growth; a large admitted prompt has already consumed Prefill compute and cache capacity. For multi-turn sessions, define whether old turns remain resident, are recomputed, are summarized, or are evicted.

Evaluate Cache Changes with Quality and SLO Gates

A cache experiment needs a stable checkpoint, engine version, hardware topology, request trace, and decoding configuration. Without those controls, a throughput difference cannot be attributed to the cache strategy.

Report percentiles, not only averages. A high prefix-cache hit rate can coexist with poor P95 TTFT if large prefixes are evicted or remote blocks arrive late.

Failure Modes and Security Boundaries

KV state is derived from user input and may retain sensitive information. Treat cache memory, hashes, metadata, transfer paths, and timing behavior as protected data.

Do not log raw prompts, token IDs, salts, or cache contents merely to debug hit rates. Use opaque identifiers, aggregate metrics, access controls, and bounded retention.

Production Decision Checklist

Before enabling or changing a KV cache strategy:

• confirm the exact model, tokenizer, adapter, engine, and cache dtype;
• calculate bytes per token from KV heads and retained-token distributions;
• reserve non-cache VRAM and fragmentation headroom;
• separate Prefill, Decode, reuse, and transfer measurements;
• define allocation, fork, free, eviction, invalidation, and rollout behavior;
• isolate cache reuse by tenant or explicit trust group;
• run output-quality regression for quantization, eviction, or compression;
• canary the change with rollback to the previous cache generation.

Sources and Related Reading

Primary technical sources:

• Hugging Face: Caching — per-layer K/V reuse and cache update semantics.
• Hugging Face: Cache strategies — Dynamic, Static, Offloaded, and Quantized Cache trade-offs.
• PagedAttention paper — paged KV management and workload-bounded throughput results.
• vLLM prefix caching design — block identity, eviction, LoRA/multimodal identity, and cache salt.
• vLLM Automatic Prefix Caching — workloads and Prefill-only performance boundary.
• KIVI paper — asymmetric KV quantization under stated experiments.
• TensorRT-LLM KV cache reuse — block reuse, eviction, and host offload boundaries.

QubitTool topic boundaries:

• LLM Inference — the complete request-to-token execution path.
• Disaggregated LLM Serving — KV transfer between Prefill and Decode pools.
• Transformer Architecture — attention, masking, and model families.
• Continuous Batching — scheduler behavior around active sequences.
• Quantization — lower-precision model and state trade-offs.