What is Speculative Decoding?
Updated on
Speculative Decoding is an autoregressive inference method that proposes multiple future Tokens with a cheaper mechanism, scores them in parallel with the target model, and accepts or corrects them under a defined verification rule.
Quick Facts
| Created | Introduced in 2022-2023 by independent Google Research and DeepMind work |
|---|---|
| Specification | Official Specification |
How It Works
Preserve the Target Distribution
For greedy decoding, a candidate is retained only while it matches the target's selected Token. For stochastic decoding, standard speculative sampling accepts a drafted Token with probability min(1, p(x)/q(x)), where p and q are target and draft probabilities at the same prefix; after rejection it samples from the normalized residual max(0, p-q). Keeping the prefix, probability and RNG contracts aligned is essential. A simple equality check is not a distribution-preserving replacement for sampled decoding.
Model the Real Speedup
Acceptance rate alone is insufficient: expected accepted length, proposer latency, target verification latency, KV memory, Sampling overhead and committed Tokens per round determine the result. Short outputs pay setup cost without many Decode steps to amortize it. At high Concurrency, Continuous Batching may already use the target's spare compute, so verification of wider drafts can reduce Throughput. Current [vLLM documentation](https://docs.vllm.ai/en/latest/features/speculative_decoding/) therefore positions the technique primarily for medium-to-low-QPS, memory-bound workloads and requires workload-specific benchmarking.
Validate Before Rollout
Pin target, draft, Tokenizer, Runtime, precision, sampling parameters and draft length. Compare against non-speculative decoding over the same prompts, output limits, arrival distribution and Concurrency. Record accepted Tokens per round and by draft position, proposer and verifier time, TTFT, TPOT, end-to-end latency, Throughput, memory and failures. Run greedy equality tests where applicable, stochastic distribution tests, task-quality and structured-output gates, then keep automatic fallback when the measured speedup or compatibility boundary fails.
Key Characteristics
- Uses a cheaper model, auxiliary head or retrieval mechanism to propose future Tokens
- Scores multiple candidate positions in one target-model verification pass
- Commits only an accepted prefix plus a target-derived correction or bonus Token
- Preserves the target distribution only when the specified rejection sampler is implemented correctly
- Trades fewer sequential target calls for proposer, verification, memory and scheduling overhead
- Produces workload-dependent gains that can disappear at short outputs or high Concurrency
Common Use Cases
- Reducing TPOT for low-to-medium-QPS interactive generation
- Accelerating long Decode workloads with a well-matched draft mechanism
- Using n-gram or suffix proposals for repetitive text and code
- Evaluating native multi-Token prediction or learned draft heads
- Canary-testing Decode optimizations without changing the public target model
Example
Loading code...Frequently Asked Questions
Does Speculative Decoding return exactly the same output?
Standard verification preserves the target probability distribution within numerical precision. Greedy equality can be tested directly, but sampled runs need not be token-for-token identical because RNG consumption, floating-point kernels and Batch composition can differ.
What is the difference between acceptance rate and acceptance length?
Acceptance rate is accepted drafted Tokens divided by proposed Tokens. Acceptance length is how many drafted Tokens a round commits before rejection. End-to-end speed depends more directly on committed progress relative to proposer and verifier cost, so report both.
Does Speculative Decoding reduce TTFT?
Usually its direct target is Decode latency after the first Token. A separate draft Prefill or setup step can leave TTFT unchanged or make it worse, especially for short responses. Measure TTFT independently rather than inferring it from faster TPOT.
Why can Speculative Decoding hurt high-concurrency throughput?
Wide verification consumes compute and memory that Continuous Batching could use for real requests. As Concurrency rises and the target becomes compute-bound, extra proposal and verification work can outweigh fewer sequential Decode rounds.
How should a draft mechanism be selected?
Measure proposer cost and accepted length on the real domain. Check vocabulary and Runtime compatibility, memory, draft length, sampling behavior and deployment complexity. A smaller parameter count alone does not prove that a draft model is cheap or well aligned.