LLM Inference: Request Flow, Metrics, and Optimization

Direct Answer

LLM inference is more than one model forward pass. A production request is validated, rendered through a versioned chat template, tokenized, queued, scheduled for Prefill, decoded token by token, stopped, detokenized, returned or streamed, and accounted for. The model parameters normally remain unchanged during that request, but the request still depends on mutable serving state such as KV cache, scheduler queues, adapters, and generation settings.

Measure the path from a declared boundary. Client-visible Time to First Token (TTFT) may include gateway, network, queue, tokenization, and Prefill time; an engine metric may begin later. Time Per Output Token (TPOT), Inter-token Latency (ITL), end-to-end latency, token throughput, and SLO-qualified goodput answer different questions. Optimize only after the model revision, workload distribution, cache state, and measurement boundary are fixed.

Trace the End-to-End Request Path

The request changes shape several times before the user sees text:

Each boundary can fail or add latency:

The GPU phase is central, but a GPU-only trace cannot explain gateway delay, client buffering, or a retry that duplicated work.

Freeze the Inference Contract Before Tuning

Performance and output comparisons are invalid if the execution identity changes between runs. Record an immutable contract rather than a model alias alone:

{
  "modelId": "provider/model-name",
  "modelRevision": "immutable-revision",
  "tokenizerRevision": "immutable-revision",
  "chatTemplateVersion": "chat-v4",
  "adapterId": "tenant-a/support-lora@revision",
  "engineVersion": "pinned-version",
  "generation": {
    "maxNewTokens": 256,
    "temperature": 0.2,
    "topP": 0.95,
    "seed": 42,
    "stop": ["</final>"]
  },
  "stream": true
}

Also record quantization format, weight and KV dtypes, parallelism topology, attention backend, speculative configuration, prefix-cache policy, and hardware. A fixed seed narrows one source of variation; it does not guarantee bitwise equivalence across software versions, kernels, devices, or distributed schedules.

Treat Templates and Tokenization as Model Inputs

A chat request is not sent to the model as a JSON object. The serving layer serializes roles, tool calls, system instructions, and separators through a chat template, then a tokenizer maps that text to IDs. A template or tokenizer revision can change:

• Prompt length and context-window fit;
• Stop-token behavior and visible control tokens;
• Prefix-cache identity and hit rate;
• Tool-call syntax;
• Output distribution, even when weights are unchanged.

Validate the fully rendered prompt and token count in staging. Do not log raw prompts by default; store redacted metadata, hashes, revision IDs, and length distributions when content is sensitive.

Prefill and Decode Are Different, Not Absolute, Bottlenecks

Prefill processes known input positions in parallel and builds the initial KV cache. For a standard full-attention Transformer, the attention portion grows quadratically with prompt length, while projections and MLP work contain substantial linear terms. Sliding-window, sparse, recurrent, or hybrid architectures change that scaling.

Decode is autoregressive: each accepted token becomes input to the next step. With KV caching, the engine projects only new positions but reads retained state and model weights. Decode is frequently constrained by memory bandwidth at small batch sizes, while larger batches, MoE routing, long attention, communication, or speculative verification can shift the bottleneck.

Use a profiler and engine metrics to identify the active constraint. Labels such as "compute-bound" are hypotheses, not capacity plans.

Decoding Strategy Changes Behavior and Cost

The model produces logits; the decoding strategy selects tokens and determines how many candidate sequences or verification passes are executed. The Hugging Face generation strategy guide distinguishes greedy selection, sampling, beam search, and custom generation loops.

• Greedy decoding chooses the highest-scoring next token.
• Sampling draws from a transformed distribution controlled by parameters such as temperature, top-p, and top-k.
• Beam search retains multiple candidate sequences and can multiply state and compute.
• Speculative decoding proposes tokens with another method and verifies them with the target model; acceptance rate and implementation determine the speedup.

Generation settings are part of the product contract. Benchmarking greedy output and deploying sampling does not measure the deployed workload. A faster configuration is not equivalent if it changes structured-output validity, task success, safety behavior, or finish reasons.

Scheduling Converts Requests into a Shared Workload

Continuous batching lets an engine admit and retire sequences at iteration boundaries instead of waiting for a fixed batch to finish. This can improve aggregate utilization, but it does not eliminate trade-offs:

• Long Prefill work can delay active Decode sequences;
• A large active batch can improve throughput while increasing TPOT;
• Preemption may free capacity but add recomputation or transfer;
• Priority can protect one tenant while starving another;
• Prefix hits change the actual number of Prefill tokens computed;
• Chunked Prefill can reduce blocking but changes scheduling and latency distributions.

In current vLLM V1, Chunked Prefill is enabled whenever possible. The scheduler prioritizes pending Decode work and spends the remaining max_num_batched_tokens budget on Prefill, splitting a prompt when necessary. This is versioned engine behavior, not a universal default: smaller token budgets can favor ITL, while larger budgets can favor TTFT or throughput, so preserve the engine release and benchmark the workload distribution.

Admission should use token and memory demand, not request count alone. Bound prompt tokens, requested output tokens, parallel samples, adapter residency, and tenant concurrency before work reaches the GPU.

Define Metrics at Explicit Boundaries

Current vLLM production metrics expose queue, Prefill, Decode, TTFT, request-level TPOT, ITL, end-to-end, token-count, preemption, and KV-usage signals. Other engines may define or name intervals differently.

Report p50, p90, p95, or p99 as appropriate, not only a mean. State whether timestamps come from the client, gateway, API process, scheduler, or model executor. For non-streaming responses, client-side TTFT is not directly observable as a token delivery interval.

Run a Boundary-Aware Trace Calculation

This dependency-free Go example calculates client-observed request metrics. TPOT uses the intervals after the first delivered token; a one-token response has no such interval and returns zero.

package main

import (
	"fmt"
	"math"
	"sort"
)

type RequestTrace struct {
	ArrivedS     float64
	FirstTokenS  float64
	LastTokenS   float64
	CompletedS   float64
	OutputTokens int
}

type RequestMetrics struct {
	TTFTMS float64
	TPOTMS float64
	E2EMS  float64
}

func (trace RequestTrace) metricsMS() (RequestMetrics, error) {
	if trace.ArrivedS < 0 ||
		trace.ArrivedS > trace.FirstTokenS ||
		trace.FirstTokenS > trace.LastTokenS ||
		trace.LastTokenS > trace.CompletedS {
		return RequestMetrics{}, fmt.Errorf("trace timestamps must be monotonic")
	}
	if trace.OutputTokens <= 0 {
		return RequestMetrics{}, fmt.Errorf("output tokens must be positive")
	}

	decodeIntervals := trace.OutputTokens - 1
	tpot := 0.0
	if decodeIntervals > 0 {
		tpot = (trace.LastTokenS - trace.FirstTokenS) / float64(decodeIntervals)
	}
	return RequestMetrics{
		TTFTMS: (trace.FirstTokenS - trace.ArrivedS) * 1000,
		TPOTMS: tpot * 1000,
		E2EMS:  (trace.CompletedS - trace.ArrivedS) * 1000,
	}, nil
}

func ordered(values []float64) []float64 {
	result := append([]float64(nil), values...)
	sort.Float64s(result)
	return result
}

func nearestRank(values []float64, percentile float64) (float64, error) {
	if len(values) == 0 || percentile <= 0 || percentile > 100 {
		return 0, fmt.Errorf("provide values and a percentile in (0, 100]")
	}
	values = ordered(values)
	index := int(math.Ceil(percentile/100*float64(len(values)))) - 1
	return values[index], nil
}

func median(values []float64) (float64, error) {
	if len(values) == 0 {
		return 0, fmt.Errorf("provide at least one value")
	}
	values = ordered(values)
	middle := len(values) / 2
	if len(values)%2 == 1 {
		return values[middle], nil
	}
	return (values[middle-1] + values[middle]) / 2, nil
}

func mustMetric(value float64, err error) float64 {
	if err != nil {
		panic(err)
	}
	return value
}

func main() {
	traces := []RequestTrace{
		{0.0, 0.6, 1.4, 1.45, 5},
		{0.2, 1.0, 2.8, 2.85, 10},
		{0.4, 1.6, 2.2, 2.25, 4},
	}

	var ttfts, tpots, e2es []float64
	minArrived := traces[0].ArrivedS
	maxCompleted := traces[0].CompletedS
	outputTokens := 0
	for _, trace := range traces {
		metrics, err := trace.metricsMS()
		if err != nil {
			panic(err)
		}
		ttfts = append(ttfts, metrics.TTFTMS)
		tpots = append(tpots, metrics.TPOTMS)
		e2es = append(e2es, metrics.E2EMS)
		minArrived = math.Min(minArrived, trace.ArrivedS)
		maxCompleted = math.Max(maxCompleted, trace.CompletedS)
		outputTokens += trace.OutputTokens
	}

	ttftP50 := mustMetric(median(ttfts))
	ttftP95 := mustMetric(nearestRank(ttfts, 95))
	tpotP95 := mustMetric(nearestRank(tpots, 95))
	e2eP95 := mustMetric(nearestRank(e2es, 95))
	duration := maxCompleted - minArrived
	if duration <= 0 {
		panic("trace window must be positive")
	}

	fmt.Printf("TTFT p50/p95: %.0f/%.0f ms\n", ttftP50, ttftP95)
	fmt.Printf("TPOT p95: %.0f ms\n", tpotP95)
	fmt.Printf("E2E p95: %.0f ms\n", e2eP95)
	fmt.Printf(
		"Aggregate output throughput: %.2f tokens/s\n",
		float64(outputTokens)/duration,
	)
}

This small sample is a calculation example, not a statistically valid benchmark. Production comparisons need enough requests for stable tails and confidence intervals.

Budget Capacity Across All Memory Consumers

Do not attribute every OOM to KV cache. A serving worker's device budget includes:

device_memory =
    model_weights
    + kv_state
    + activations_and_workspaces
    + communication_buffers
    + graph_or_compile_reservations
    + adapters
    + allocator_fragmentation
    + safety_headroom

Model quantization can free weight memory without shrinking KV state. KV quantization changes a different allocation and may affect quality or kernel support. Tensor parallelism can make a model fit while introducing communication. Offload trades device memory for transfer latency. Measure peak allocated and reserved memory under the real prompt/output distribution, concurrency, prefix reuse, adapters, and speculative branches.

Map Symptoms to the Right Control

Optimization is workload-specific. Do not copy a throughput multiplier from another model, context shape, engine release, or GPU.

Benchmark a Distribution, Not a Hero Number

A credible serving benchmark records:

1. Immutable model, tokenizer, template, adapter, engine, driver, kernel, hardware, and topology revisions;
2. Prompt and output token distributions, multimodal sizes, generation settings, and finish reasons;
3. Open-loop arrival rate or closed-loop concurrency, including burstiness and warm-up;
4. Cold versus warm model state and enabled Prefix Cache policy;
5. TTFT, TPOT, ITL, E2E, request/output throughput, errors, preemptions, and memory percentiles;
6. Quality, structured-output validity, safety, and application success;
7. SLO-qualified goodput across an offered-load sweep.

The vLLM benchmark implementation reports TTFT, TPOT, ITL, E2E, request throughput, output throughput, total-token throughput, percentiles, and optional goodput. Its CLI and metric semantics are versioned, so pin and archive the exact tool revision with results. The DistServe paper defines goodput around the maximum request rate served within TTFT and TPOT constraints; its reported multipliers remain specific to the evaluated systems and workloads.

Preserve Quality and Reproducibility

Inference optimization can change outputs even when the API still returns HTTP 200:

• Weight, activation, or KV precision changes can alter logits;
• A new tokenizer or template changes model input;
• Sampling and speculative acceptance introduce path-dependent behavior;
• Kernel, compiler, parallel reduction, and hardware changes can alter floating-point results;
• Stop handling can truncate content or expose control tokens;
• Scheduler changes can surface race, timeout, or cancellation behavior.

PyTorch's reproducibility guidance explicitly states that complete reproducibility is not guaranteed across releases, commits, platforms, or CPU/GPU execution, even with identical seeds. Use a pinned environment and deterministic controls for debugging, then run task-level and output-distribution evaluations for production acceptance.

Enforce Failure and Security Boundaries

• Authorize model and adapter access before queue admission.
• Isolate Prefix/KV reuse by compatible revision, modality, and tenant scope.
• Enforce prompt, output, parallel-sample, timeout, and cost limits server-side.
• Cancel GPU work when the client disconnects unless an explicit asynchronous job contract says otherwise.
• Make retries idempotent at the billing and job layer; a dropped stream may have already consumed tokens.
• Record finish reason, token counts, revision identity, and trace IDs without logging sensitive prompt content by default.
• Test malformed role sequences, oversized requests, stop-string edges, Unicode chunk boundaries, cancellations, and partial failures.

Production Decision Checklist

Before rollout:

• Is the execution identity immutable and visible in traces?
• Are client and engine timing boundaries documented?
• Does the load test reproduce prompt/output lengths, arrival rate, burstiness, and cache state?
• Are p95/p99 TTFT, TPOT/ITL, E2E, goodput, errors, preemptions, and memory within gates?
• Do task quality, structured outputs, safety, and finish reasons match the baseline?
• Are tenant isolation, quotas, cancellation, retry, and billing behavior tested?
• Can the previous engine, model, template, tokenizer, adapter, and scheduler configuration be restored independently?

Sources and Related Reading

• Hugging Face: Generation Strategies — greedy, sampling, beam, and custom decoding behavior.
• Hugging Face: Generation Features — streaming behavior and application-facing generation features.
• vLLM: Production Metrics — queue, Prefill, Decode, TTFT, TPOT, ITL, E2E, token, and cache metrics.
• vLLM: Optimization and Tuning — current V1 preemption, Chunked Prefill, and scheduling trade-offs.
• vLLM: Metrics Design — event boundaries, preemption, and interval calculations.
• DistServe — TTFT/TPOT SLOs and goodput under Prefill/Decode disaggregation.
• PyTorch: Reproducibility — seed, deterministic algorithm, platform, and version limits.
• KV Cache for LLM Inference — state capacity, Prefix Cache, paging, quantization, and isolation.
• Disaggregated LLM Serving — when and how to separate Prefill and Decode pools.
• Model Quantization Guide — weight precision, format, hardware, and quality trade-offs.