What is Prefill?
Prefill is the LLM inference phase that processes a request's uncached input tokens, produces the first output-token state, and builds the key-value cache or other model state reused by later decoding.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
Measurement Boundary
Prefill latency is an engine-side interval, commonly measured from the request's first scheduling event until its first output-producing model step. The [vLLM metrics design](https://docs.vllm.ai/en/stable/design/metrics/) records Queue, Scheduled and New Tokens events so those intervals can be separated. TTFT is broader because a client may also observe request transport, authentication, admission, queueing, tokenization and response delivery. The exact event definitions vary by benchmark and runtime, so record them rather than subtracting undocumented timestamps. A client trace alone usually cannot isolate kernel-level prefill time.
State and Scheduling
Uncached prompt positions create reusable attention state for later Decode. Prefix caching can reduce the number of positions processed, but Cache Lookup, isolation and misses remain part of the system. [SARATHI](https://arxiv.org/abs/2308.16369) studies Chunked Prefill that interleaves prompt chunks with Decode work; its measured gains are specific to the evaluated models and hardware. [DistServe](https://arxiv.org/abs/2401.09670) moves the resulting KV state to a Decode Worker, which can improve SLO-qualified capacity for some workloads while adding routing and transfer cost.
Benchmark Contract
A reproducible Prefill result identifies model and Tokenizer revisions, uncached and cached-prefix Token counts, preprocessing, Batch and arrival shape, scheduler, hardware, precision, Cache state and timing boundary. The [AIPerf metrics reference](https://docs.nvidia.com/aiperf/reference/ai-perf-metrics-reference) separates record, aggregate and derived metrics. Report p50, p95 and p99 Latency with input-token throughput, Queue Time, TTFT, Decode Latency, failures and output quality. Do not compare a Warm Cache Hit with a Cold Prompt or divide all input Tokens by TTFT and call the result pure model throughput without stating the approximation.
Key Characteristics
- Processes uncached input positions before steady token-by-token generation
- Builds KV cache or architecture-specific state consumed by decode
- Can run as one pass, scheduler-sized chunks, or a separate serving stage
- Contributes to TTFT but excludes some client, gateway, queue and delivery time
- Competes with decode for compute, memory and scheduling capacity when colocated
- Must be reported with prompt shape, cache state, load and event definitions
Common Use Cases
- Separating queue delay from prompt-processing time in a slow chat request
- Sizing long-context, RAG and multimodal workloads by uncached input tokens
- Testing whether prefix caching removes real prefill work without cross-tenant reuse
- Choosing chunked, colocated or disaggregated prefill under TTFT and ITL SLOs
- Detecting regressions after model, tokenizer, scheduler, kernel or hardware changes
Example
Loading code...Frequently Asked Questions
Is prefill the same as tokenization or TTFT?
No. Tokenization maps input to token IDs. Prefill runs the model over uncached input positions and creates reusable state. TTFT is a client-observed interval that can additionally include network transit, authentication, queueing, preprocessing and delivery of the first non-empty output.
Does prefill always process the entire prompt at once?
No. A cold request may require all prompt positions, a prefix-cache hit may require only the unmatched suffix, and a scheduler may divide a long prefill into chunks. Report uncached token count and scheduling mode so the result can be interpreted.
Is prefill always compute-bound?
No universal bottleneck applies. Prefill often exposes high matrix-compute parallelism, but attention shape, model architecture, batch size, memory traffic, communication, cache behavior and multimodal encoding can shift the limiting resource.
How should prefill performance be measured?
Use engine timestamps from scheduling to the first output-producing step, then report latency percentiles and uncached input-token throughput. Keep queue time, TTFT and network delivery separate, and pin model, tokenizer, cache state, workload, runtime and hardware.
When do chunked or disaggregated prefill help?
They help only when measured interference or phase imbalance prevents the workload from meeting TTFT and inter-token latency objectives. Chunking trades prompt completion time against decode fairness; disaggregation adds KV-transfer and routing cost. Compare both with a colocated baseline.