What is Latency?
Latency is the elapsed wall-clock time between two explicitly named events in an AI system, such as client request start and first content Token, final response, or completed tool result.
How It Works
Define the Measurement Boundary
Name the clock, start event and terminal event before collecting data. Use a monotonic clock within one process; distributed spans require synchronized timestamps or durations measured locally at each hop. Separate client, Gateway, Queue, Prefill, Decode, Retrieval, Tool and network intervals. These spans can overlap, so their individual durations should not be added unless the trace proves they are sequential. Keep successful, failed, timed-out, cancelled and truncated requests visible rather than silently dropping slow outcomes.
Diagnose the Right Stage
High TTFT can come from Queue wait, cold model loading, long-prompt Prefill, Cache misses or network delay. High TPOT or ITL points toward Decode scheduling, memory bandwidth, growing KV Cache, preemption or an overloaded Batch. High end-to-end latency with healthy model metrics may instead come from output length, retrieval, tool execution or post-processing. [NVIDIA's metrics guide](https://docs.nvidia.com/nim/benchmarking/llm/1.0.0/metrics.html) also warns that benchmark tools use different event and aggregation conventions, so names alone do not make results comparable.
Turn Percentiles into an SLO
Compute percentiles from request-level samples or merge compatible Histogram buckets; never average p95 values from different Replicas. Slice results by workload shape and version before comparing them, then raise Offered Load until tail latency or errors exceed the objective. An SLO should state the eligible request population, metric boundary, percentile, threshold and evaluation window. Pair it with Goodput so a configuration is not called faster when it serves more traffic by violating latency, correctness, quality or safety gates.
Key Characteristics
- Measures time only after start, end, clock and request population are declared
- Separates client-perceived, server, Queue, Prefill, Decode and dependency boundaries
- Requires request-level distributions such as p50, p95 and p99 instead of an average alone
- Changes with input and output length, Cache state, arrival pattern, Concurrency and failures
- Can worsen while aggregate Throughput rises because batching and queueing trade capacity for wait time
- Becomes operationally useful when tied to SLO-qualified Goodput and trace evidence
Common Use Cases
- Setting separate TTFT, TPOT and end-to-end objectives for a streaming assistant
- Finding whether a regression originates in Queue, Prefill, Decode, Retrieval or a Tool
- Comparing serving configurations under the same model, workload and Offered Load
- Sizing capacity from tail latency and Goodput rather than average response time
- Verifying canary releases, autoscaling and timeout policies before wider rollout
Example
Loading code...Frequently Asked Questions
What is the difference between TTFT, TPOT and end-to-end latency?
TTFT ends when the first content Token arrives. TPOT averages the intervals after that first Token for one request under a declared convention. End-to-end latency ends when the final usable response arrives. Each answers a different user or system question.
Why should LLM latency use p95 or p99 instead of only an average?
An average can remain stable while a minority of requests suffer long queues, retries, cold starts or dependency stalls. Tail percentiles expose those experiences, but they still need the request population, window and outcome policy to be meaningful.
Can server latency be compared directly with browser latency?
No. Server timing may exclude client networking, gateway work, connection setup and rendering. Compare values only when their start and end events match, or use a distributed trace to retain each boundary separately.
How should cancelled, failed and timed-out requests affect latency reports?
Report them as separate outcomes and include them in the attempted-request denominator for reliability or Goodput. Dropping them from the dataset can make an overloaded service look artificially fast.
Does reducing average latency prove a serving change is better?
No. Verify tail latency, errors, Goodput, output quality, safety and cost under the same workload and Offered Load. A change can lower the mean by rejecting hard requests, shortening outputs or shifting work outside the measured boundary.