What is KV Cache?

KV Cache (Key-Value Cache) is per-layer inference state that stores the attention Key and Value tensors for retained tokens, allowing autoregressive decoding to reuse them instead of projecting the full prefix again.

Quick Facts

Full NameKey-Value Cache
CreatedUsed as incremental attention state in autoregressive Transformer decoding; implementations evolved after the 2017 Transformer architecture

How It Works

During prefill, a causal Transformer creates K/V state for prompt tokens in each attention layer. During decode, it computes Q/K/V for the newest token, reads retained K/V, and appends new state. Caching removes repeated historical projection, but standard attention work and reads still grow with retained sequence length. A basic decoder-only estimate is 2 × layers × KV heads × head dimension × bytes per element × resident token slots. GQA/MQA, sliding windows, quantization, paging, prefix sharing, offload, speculative branches, and implementation layout change physical allocation. Cross-request reuse also requires compatible model, tokenizer, adapter, template, modality, dtype, and trust scope.

Key Characteristics

  • Stores Key and Value tensors for retained tokens separately in each attention layer
  • Removes repeated historical projections while attention still reads the retained cache
  • Logical bytes scale with layers, KV heads, head dimension, dtype, and resident token slots
  • Can use dynamic, static, paged, sliding, offloaded, quantized, or shared-prefix layouts
  • Requires a compatibility identity before blocks can be reused across requests
  • May change output semantics when quantization, eviction, compression, or incompatible reuse is introduced

Common Use Cases

  1. Accelerating LLM text generation in production inference servers
  2. Planning resident-token capacity and admission limits for concurrent requests
  3. Reusing authorized compatible prefixes in multi-turn and document workloads
  4. Reducing GPU pressure through paging, lower precision, or memory-tier offload
  5. Measuring TTFT, TPOT, evictions, transfers, and quality during serving changes

Example

loading...
Loading code...

Frequently Asked Questions

What is KV Cache in Transformer models?

KV Cache is per-layer inference state containing Key and Value tensors for retained tokens. During autoregressive decode, the newest token supplies a new Query and adds new K/V, while historical K/V are reused. Queries and attention scores are not cached because each position produces a new Query and therefore a new score vector.

How does KV Cache affect memory usage?

For a basic decoder-only layout, estimate bytes as batch_size × sequence_length × n_layers × 2 (K and V) × n_kv_heads × head_dim × bytes_per_element. Use KV heads, not Query heads, for GQA/MQA models. This excludes weights, activations, temporary workspaces, allocator fragmentation, paging metadata, and implementation-specific layout, so validate the estimate against the exact checkpoint and serving engine.

What are Multi-Query Attention and Grouped-Query Attention?

MQA shares one Key-Value head across Query heads, while GQA shares a smaller set of KV heads among Query groups. Relative to full multi-head attention at the same dimensions and dtype, the cache reduction ratio is approximately Query heads divided by KV heads. Quality is a property of the trained architecture and workload, not a guaranteed minimal trade-off.

Can KV Cache be quantized to save memory?

Yes, when the exact checkpoint, serving engine, and hardware kernels support the selected cache dtype. The nominal bit-width ratio overstates realized savings because scales, padding, metadata, and layout add overhead. Quality and latency effects depend on model, layer sensitivity, calibration, and workload, so benchmark memory, throughput, and task quality together.

What is prefix caching and how does it relate to KV Cache?

Prefix caching reuses compatible K/V blocks for a shared token prefix across requests. It can reduce repeated prefill and TTFT, but it does not reduce decode work for new output tokens. Benefits depend on reused-token volume, locality, and eviction; model, tokenizer, adapter, template, modality, dtype, tenant isolation, permissions, and retention must all remain compatible.

Related Terms

Attention Mechanism

Attention Mechanism is a differentiable operation that computes data-dependent weights over permitted representations and uses them to form an output. Its behavior depends on the Query, Key, and Value projections, masking, normalization, learned parameters, and the surrounding model.

Transformer

Transformer is a neural-network architecture first described in the 2017 paper "Attention Is All You Need." Its common variants combine attention, feed-forward layers, residual connections, normalization, and positional information to process token or other sequence representations.

Prefill

Prefill is the LLM inference phase that processes the full input prompt in parallel and produces the initial key-value cache before token-by-token decoding begins.

Decode Phase

Decode Phase is the LLM inference stage that generates output one token at a time using the KV cache created during prefill.

Related Articles

KV Cache for LLM Inference: Memory, Reuse, and Tuning

Learn how KV cache works in LLM inference, calculate memory per token, compare dynamic, static, paged, offloaded, and quantized caches, and tune prefix reuse.

2026-04-06

AI Accelerator Selection: GPUs, TPUs, and Custom Silicon

Select AI accelerators with a reproducible workload contract. Compare GPUs, TPUs, and custom silicon using memory, latency, goodput, software fit, migration risk, and complete TCO.

2026-05-22

Ollama vs vLLM vs llama.cpp: Local LLM Deployment Guide

Compare Ollama, vLLM, and llama.cpp for local LLM deployment. Learn workload fit, concurrency, API compatibility, fair benchmarks, migration, and production controls.

2026-05-16