What is TTFT?
TTFT (Time to First Token) is the client-observed elapsed time from sending a streaming LLM request until receiving the first non-empty output token or token-bearing chunk.
Quick Facts
| Full Name | Time to First Token |
|---|---|
| Specification | Official Specification |
How It Works
Decompose Before Optimizing
A high TTFT can come from network and gateway work, a cold model or adapter, queue saturation, tokenization and multimodal preprocessing, uncached Prefill, Cache Lookup or Transfer, or first-output delivery. The [vLLM metrics design](https://docs.vllm.ai/en/stable/design/metrics/) exposes Queue, Prefill and first-output events that help decompose it. Collect timestamps at trusted boundaries and correlate them with Prompt Tokens, Cached Tokens, Queue Depth, Model Residency and errors. Shortening a Prompt will not fix queue saturation; adding a Prefix Cache will not fix a Cold Worker or slow client network.
Report a Distribution
Report p50, p95 and p99 TTFT for a declared traffic distribution rather than one average or best run. The [AIPerf reference](https://docs.nvidia.com/aiperf/reference/ai-perf-metrics-reference) defines TTFT from request start to the first non-empty response and keeps it as a per-request distribution. Preserve input-length buckets, Cache-Hit state, concurrency, arrival pattern, region, model and Runtime revision, hardware and success criteria. Separate Cold Starts from Steady State and aborted or failed requests from successful responses. Closed-loop load can hide the wait that real arrivals experience, so state how load was generated.
Use TTFT With Other Metrics
TTFT covers only response startup. Pair it with ITL or TPOT for streaming cadence, end-to-end latency for completion, output length, errors, quality and system throughput. A service can improve TTFT by prioritizing short prompts while starving long requests, or raise throughput by delaying individual requests in larger batches. Goodput counts only requests that satisfy the full quality and latency contract.
Key Characteristics
- Starts at a declared client or server request event and ends at first non-empty output
- Includes more than Prefill unless the metric explicitly uses engine-only events
- Requires streaming; a non-streaming response usually exposes only completion latency
- Changes with queueing, cold state, input length, cache hits, routing and network path
- Should be reported as a distribution segmented by workload and cache state
- Does not describe post-first-token cadence, answer quality or service capacity
Common Use Cases
- Setting a response-start SLO for interactive chat, coding or search
- Separating queue saturation from long-prompt Prefill
- Comparing warm, cold and prefix-cache-hit behavior
- Detecting regressions after routing, batching, model or hardware changes
- Evaluating whether a faster first response still preserves fairness and answer quality
Example
Loading code...Frequently Asked Questions
Is TTFT the same as Prefill latency?
No. Prefill latency is an engine phase. Client-observed TTFT can also include transport, authentication, admission, queueing, preprocessing, tokenization, the first output-producing step and response delivery. Compare values only after aligning their event boundaries.
Is TTFT meaningful for a non-streaming response?
Usually not as a separate user-visible metric because the client receives the response only after completion. Some servers expose an internal first-token timestamp, but that is not the same observation as a streaming client TTFT and must be labeled separately.
What is a good TTFT for an LLM?
There is no universal threshold. Set an SLO from the product's interaction pattern and risk, then measure representative input lengths, cache states, regions and load. Report tail percentiles and successful-task quality instead of adopting a generic number.
Why can TTFT rise while Decode speed stays stable?
Queueing, cold model load, longer uncached prompts, slower preprocessing or KV transfer can delay the first output without changing steady-state Decode. Segment those intervals before tuning the model or scheduler.
Can a low TTFT prove that an LLM service is fast?
No. It proves only that output started quickly under the measured conditions. The stream may still have high ITL, long completion time, errors, truncated output, low quality or poor aggregate throughput. A production decision needs the complete scorecard.