What is Decode Phase?
Decode Phase is the autoregressive LLM inference stage after Prefill in which accepted output tokens extend the sequence and reusable model state until a stop condition is reached.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
Measurement Boundary
Client-observed Decode Duration is commonly the interval from the first non-empty streamed output to the final non-empty output. The [AIPerf metrics reference](https://docs.nvidia.com/aiperf/reference/ai-perf-metrics-reference) and [vLLM metrics design](https://docs.vllm.ai/en/stable/design/metrics/) document related but tool-specific event boundaries. Request-level TPOT divides that interval by output_tokens - 1, because the first Token ends TTFT and only the remaining Token intervals lie inside Decode Duration. ITL instead measures individual gaps between output events. Bundled Chunks, reasoning-only events, retries and preemption can make these metrics diverge.
Scheduling and State
Continuous Batching combines Decode positions from multiple active sequences and admits new work at scheduler boundaries. It can increase aggregate utilization while lowering per-user Token rate or increasing Jitter. Preemption may evict or recompute state, and [DistServe](https://arxiv.org/abs/2401.09670) shows why a disaggregated design must account for KV transfer and placement. Every comparison should include Queueing, Cache Pressure, Preemption, Batch Occupancy and Cancellation rather than attributing all delay to the model Kernel.
Optimization and Quality
Quantization, optimized Attention, KV-cache layout, speculative decoding and parallelism can change Decode cost, but speed is valid only when output semantics remain acceptable. Pin model, tokenizer, runtime, precision, sampling policy and stop conditions; compare identical input/output distributions; report p50/p95/p99 TPOT or ITL, per-user output TPS, aggregate throughput, errors and quality. A faster stream that truncates, repeats, violates a schema or fails Tool Calls is not a successful optimization.
Key Characteristics
- Extends each request from an accepted prefix until a terminal outcome
- Reuses and appends KV Cache or architecture-specific recurrent state
- Has a sequential dependency per request but can be batched across requests
- May accept multiple tokens per model step and emit multiple tokens per chunk
- Is measured with Decode Duration, ITL, TPOT and qualified output TPS
- Must be evaluated with quality, stop behavior, failures and cache pressure
Common Use Cases
- Diagnosing slow or uneven streaming after the first output arrives
- Comparing continuous batching and speculative decoding under the same SLO
- Sizing KV-cache capacity for long outputs and concurrent sequences
- Separating first-output responsiveness from sustained generation speed
- Detecting quality or termination regressions after runtime optimization
Example
Loading code...Frequently Asked Questions
Does the Decode Phase always generate exactly one token per step?
Conventional autoregressive decoding has a sequential dependency at each position, but speculative and multi-token methods can accept several tokens in one target-model step. A streaming transport may also bundle multiple tokens into one chunk, so steps, chunks and tokens are different units.
Is Decode Duration the same as end-to-end latency?
No. Client-observed Decode Duration starts with the first non-empty output and ends with the last. End-to-end latency also includes request transport, queueing, preprocessing, Prefill and first-token generation.
What is the difference between TPOT and ITL?
TPOT is one request-level average, usually `(end-to-end latency - TTFT) / (output tokens - 1)`. ITL records each gap between output events. They differ when chunks contain multiple tokens, timing is uneven, or aggregate weighting differs.
Why can higher aggregate throughput reduce per-user Decode speed?
A scheduler may batch more active sequences to amortize weight reads and raise total output tokens per second. Each sequence then receives less frequent service, so per-user TPOT or token delivery jitter can worsen even as the server completes more total work.
How should a Decode optimization be accepted?
Replay the same model, tokenizer, request distribution, output-length policy and quality tests. Compare tail TPOT or ITL, per-user output TPS, aggregate throughput, errors, preemption, memory and accepted-task quality rather than a single peak token rate.