Direct Answer

LLM inference is more than one model forward pass. A production request is validated, rendered through a versioned chat template, tokenized, queued, scheduled for Prefill, decoded token by token, stopped, detokenized, returned or streamed, and accounted for. The model parameters normally remain unchanged during that request, but the request still depends on mutable serving state such as KV cache, scheduler queues, adapters, and generation settings.

Measure the path from a declared boundary. Client-visible Time to First Token (TTFT) may include gateway, network, queue, tokenization, and Prefill time; an engine metric may begin later. Time Per Output Token (TPOT), Inter-token Latency (ITL), end-to-end latency, token throughput, and SLO-qualified goodput answer different questions. Optimize only after the model revision, workload distribution, cache state, and measurement boundary are fixed.

Trace the End-to-End Request Path

The request changes shape several times before the user sees text:

flowchart LR A["API request"] --> B["Validate and authorize"] B --> C["Render chat template"] C --> D["Tokenize"] D --> E["Queue and admit"] E --> F["Prefill"] F --> G["Initial KV state"] G --> H["Decode and sample"] H --> I{"Stop?"} I -->|"No"| H I -->|"Yes"| J["Detokenize and deliver"] J --> K["Finish reason, usage, trace"]

Each boundary can fail or add latency:

Stage Input Output Common failure signal
API and policy Messages, model alias, generation options Authorized normalized request 4xx, quota rejection, tenant mismatch
Template and tokenization Messages plus template revision Token IDs, masks, stop token IDs Context overflow, tokenizer drift, malformed role order
Queue and admission Token demand, priority, cache demand Scheduled sequence Queue tail, starvation, rejected capacity
Prefill Input tokens First logits and initial KV state High prompt latency, head-of-line blocking
Decode Latest token and retained state Next-token logits and appended state High ITL, preemption, OOM
Post-processing Token IDs and finish reason Text chunks and usage Broken Unicode boundary, stop mismatch, truncated stream

The GPU phase is central, but a GPU-only trace cannot explain gateway delay, client buffering, or a retry that duplicated work.

Freeze the Inference Contract Before Tuning

Performance and output comparisons are invalid if the execution identity changes between runs. Record an immutable contract rather than a model alias alone:

json 
{
  "modelId": "provider/model-name",
  "modelRevision": "immutable-revision",
  "tokenizerRevision": "immutable-revision",
  "chatTemplateVersion": "chat-v4",
  "adapterId": "tenant-a/support-lora@revision",
  "engineVersion": "pinned-version",
  "generation": {
    "maxNewTokens": 256,
    "temperature": 0.2,
    "topP": 0.95,
    "seed": 42,
    "stop": ["</final>"]
  },
  "stream": true
}

Also record quantization format, weight and KV dtypes, parallelism topology, attention backend, speculative configuration, prefix-cache policy, and hardware. A fixed seed narrows one source of variation; it does not guarantee bitwise equivalence across software versions, kernels, devices, or distributed schedules.

Treat Templates and Tokenization as Model Inputs

A chat request is not sent to the model as a JSON object. The serving layer serializes roles, tool calls, system instructions, and separators through a chat template, then a tokenizer maps that text to IDs. A template or tokenizer revision can change:

  • Prompt length and context-window fit;
  • Stop-token behavior and visible control tokens;
  • Prefix-cache identity and hit rate;
  • Tool-call syntax;
  • Output distribution, even when weights are unchanged.

Validate the fully rendered prompt and token count in staging. Do not log raw prompts by default; store redacted metadata, hashes, revision IDs, and length distributions when content is sensitive.

Prefill and Decode Are Different, Not Absolute, Bottlenecks

Prefill processes known input positions in parallel and builds the initial KV cache. For a standard full-attention Transformer, the attention portion grows quadratically with prompt length, while projections and MLP work contain substantial linear terms. Sliding-window, sparse, recurrent, or hybrid architectures change that scaling.

Decode is autoregressive: each accepted token becomes input to the next step. With KV caching, the engine projects only new positions but reads retained state and model weights. Decode is frequently constrained by memory bandwidth at small batch sizes, while larger batches, MoE routing, long attention, communication, or speculative verification can shift the bottleneck.

Dimension Prefill Decode
Primary sequence work Process input positions Produce accepted output positions
Parallelism Many known positions Dependency between accepted steps
State Creates or extends KV Reads and appends KV
User-facing signal Strong contributor to TTFT Strong contributor to ITL and TPOT
Frequent pressure Compute, long-prompt queueing Weight/KV reads, batch and scheduler contention
Important caveat Not always compute-bound Not always bandwidth-bound

Use a profiler and engine metrics to identify the active constraint. Labels such as "compute-bound" are hypotheses, not capacity plans.

Decoding Strategy Changes Behavior and Cost

The model produces logits; the decoding strategy selects tokens and determines how many candidate sequences or verification passes are executed. The Hugging Face generation strategy guide distinguishes greedy selection, sampling, beam search, and custom generation loops.

  • Greedy decoding chooses the highest-scoring next token.
  • Sampling draws from a transformed distribution controlled by parameters such as temperature, top-p, and top-k.
  • Beam search retains multiple candidate sequences and can multiply state and compute.
  • Speculative decoding proposes tokens with another method and verifies them with the target model; acceptance rate and implementation determine the speedup.

Generation settings are part of the product contract. Benchmarking greedy output and deploying sampling does not measure the deployed workload. A faster configuration is not equivalent if it changes structured-output validity, task success, safety behavior, or finish reasons.

Scheduling Converts Requests into a Shared Workload

Continuous batching lets an engine admit and retire sequences at iteration boundaries instead of waiting for a fixed batch to finish. This can improve aggregate utilization, but it does not eliminate trade-offs:

  • Long Prefill work can delay active Decode sequences;
  • A large active batch can improve throughput while increasing TPOT;
  • Preemption may free capacity but add recomputation or transfer;
  • Priority can protect one tenant while starving another;
  • Prefix hits change the actual number of Prefill tokens computed;
  • Chunked Prefill can reduce blocking but changes scheduling and latency distributions.

Admission should use token and memory demand, not request count alone. Bound prompt tokens, requested output tokens, parallel samples, adapter residency, and tenant concurrency before work reaches the GPU.

Define Metrics at Explicit Boundaries

Current vLLM production metrics expose queue, Prefill, Decode, TTFT, request-level TPOT, ITL, end-to-end, token-count, preemption, and KV-usage signals. Other engines may define or name intervals differently.

Metric Useful definition What it can hide
TTFT Request boundary to first delivered output token Gateway, tokenization, queue, Prefill, buffering
TPOT (last token - first token) / (output tokens - 1) for a request Uneven individual token gaps
ITL Each adjacent delivered-token interval Request-level average and output length
E2E latency Request boundary to terminal response Different output lengths and finish reasons
Request throughput Completed requests per second Prompt/output size and failed work
Output throughput Generated output tokens per second Latency tails and prompt processing
Goodput Requests per second satisfying declared SLOs Quality, correctness, and business success

Report p50, p90, p95, or p99 as appropriate, not only a mean. State whether timestamps come from the client, gateway, API process, scheduler, or model executor. For non-streaming responses, client-side TTFT is not directly observable as a token delivery interval.

Run a Boundary-Aware Trace Calculation

This dependency-free example calculates client-observed request metrics. TPOT uses the intervals after the first delivered token; a one-token response has no such interval and returns zero.

python 
from __future__ import annotations

from dataclasses import dataclass
from math import ceil
from statistics import median


@dataclass(frozen=True)
class RequestTrace:
    arrived_s: float
    first_token_s: float
    last_token_s: float
    completed_s: float
    output_tokens: int

    def metrics_ms(self) -> dict[str, float]:
        if not (
            0
            <= self.arrived_s
            <= self.first_token_s
            <= self.last_token_s
            <= self.completed_s
        ):
            raise ValueError("Trace timestamps must be monotonic")
        if self.output_tokens <= 0:
            raise ValueError("output_tokens must be positive")

        ttft = self.first_token_s - self.arrived_s
        e2e = self.completed_s - self.arrived_s
        decode_intervals = self.output_tokens - 1
        tpot = (
            (self.last_token_s - self.first_token_s) / decode_intervals
            if decode_intervals
            else 0.0
        )
        return {
            "ttft_ms": ttft * 1000,
            "tpot_ms": tpot * 1000,
            "e2e_ms": e2e * 1000,
        }


def nearest_rank(values: list[float], percentile: int) -> float:
    if not values or not 0 < percentile <= 100:
        raise ValueError("Provide values and a percentile in (0, 100]")
    ordered = sorted(values)
    return ordered[ceil(percentile / 100 * len(ordered)) - 1]


traces = [
    RequestTrace(0.0, 0.6, 1.4, 1.45, 5),
    RequestTrace(0.2, 1.0, 2.8, 2.85, 10),
    RequestTrace(0.4, 1.6, 2.2, 2.25, 4),
]

try:
    metrics = [trace.metrics_ms() for trace in traces]
    ttfts = [item["ttft_ms"] for item in metrics]
    tpots = [item["tpot_ms"] for item in metrics]
    e2es = [item["e2e_ms"] for item in metrics]
    duration = max(trace.completed_s for trace in traces) - min(
        trace.arrived_s for trace in traces
    )
    output_tps = sum(trace.output_tokens for trace in traces) / duration

    print(f"TTFT p50/p95: {median(ttfts):.0f}/{nearest_rank(ttfts, 95):.0f} ms")
    print(f"TPOT p95: {nearest_rank(tpots, 95):.0f} ms")
    print(f"E2E p95: {nearest_rank(e2es, 95):.0f} ms")
    print(f"Aggregate output throughput: {output_tps:.2f} tokens/s")
except ValueError as error:
    raise SystemExit(f"Invalid trace set: {error}") from error

# TTFT p50/p95: 800/1200 ms
# TPOT p95: 200 ms
# E2E p95: 2650 ms
# Aggregate output throughput: 6.67 tokens/s

This small sample is a calculation example, not a statistically valid benchmark. Production comparisons need enough requests for stable tails and confidence intervals.

Budget Capacity Across All Memory Consumers

Do not attribute every OOM to KV cache. A serving worker's device budget includes:

text 
device_memory =
    model_weights
    + kv_state
    + activations_and_workspaces
    + communication_buffers
    + graph_or_compile_reservations
    + adapters
    + allocator_fragmentation
    + safety_headroom

Model quantization can free weight memory without shrinking KV state. KV quantization changes a different allocation and may affect quality or kernel support. Tensor parallelism can make a model fit while introducing communication. Offload trades device memory for transfer latency. Measure peak allocated and reserved memory under the real prompt/output distribution, concurrency, prefix reuse, adapters, and speculative branches.

Map Symptoms to the Right Control

Symptom Evidence to inspect Candidate controls Required regression gate
TTFT tail rises Client/gateway/queue/Prefill split, prompt lengths Admission, routing, prefix reuse, chunked Prefill, more Prefill capacity Output and cache-identity parity
ITL or TPOT rises Active sequences, batch tokens, Decode time, preemption Batch budget, parallelism, lower precision, speculative decoding Token/logit or task-quality regression
Throughput plateaus Offered load, waiting requests, GPU/communication profile Continuous batching, replica count, topology, scheduler p95/p99 latency and fairness
OOM or evictions Weight/KV/workspace peaks, fragmentation Token admission, paging, quantization, offload No stale or cross-tenant reuse
Output drift Revision, template, tokenizer, generation config Roll back the changed contract component Golden and adversarial eval sets
Stream stalls Engine token event versus gateway/client delivery Buffering, backpressure, timeout, cancellation No duplicate billing or retries

Optimization is workload-specific. Do not copy a throughput multiplier from another model, context shape, engine release, or GPU.

Benchmark a Distribution, Not a Hero Number

A credible serving benchmark records:

  1. Immutable model, tokenizer, template, adapter, engine, driver, kernel, hardware, and topology revisions;
  2. Prompt and output token distributions, multimodal sizes, generation settings, and finish reasons;
  3. Open-loop arrival rate or closed-loop concurrency, including burstiness and warm-up;
  4. Cold versus warm model state and enabled Prefix Cache policy;
  5. TTFT, TPOT, ITL, E2E, request/output throughput, errors, preemptions, and memory percentiles;
  6. Quality, structured-output validity, safety, and application success;
  7. SLO-qualified goodput across an offered-load sweep.

The vLLM benchmark implementation reports TTFT, TPOT, ITL, E2E, request throughput, output throughput, total-token throughput, percentiles, and optional goodput. Its CLI and metric semantics are versioned, so pin and archive the exact tool revision with results. The DistServe paper defines goodput around the maximum request rate served within TTFT and TPOT constraints; its reported multipliers remain specific to the evaluated systems and workloads.

Preserve Quality and Reproducibility

Inference optimization can change outputs even when the API still returns HTTP 200:

  • Weight, activation, or KV precision changes can alter logits;
  • A new tokenizer or template changes model input;
  • Sampling and speculative acceptance introduce path-dependent behavior;
  • Kernel, compiler, parallel reduction, and hardware changes can alter floating-point results;
  • Stop handling can truncate content or expose control tokens;
  • Scheduler changes can surface race, timeout, or cancellation behavior.

PyTorch's reproducibility guidance explicitly states that complete reproducibility is not guaranteed across releases, commits, platforms, or CPU/GPU execution, even with identical seeds. Use a pinned environment and deterministic controls for debugging, then run task-level and output-distribution evaluations for production acceptance.

Enforce Failure and Security Boundaries

  • Authorize model and adapter access before queue admission.
  • Isolate Prefix/KV reuse by compatible revision, modality, and tenant scope.
  • Enforce prompt, output, parallel-sample, timeout, and cost limits server-side.
  • Cancel GPU work when the client disconnects unless an explicit asynchronous job contract says otherwise.
  • Make retries idempotent at the billing and job layer; a dropped stream may have already consumed tokens.
  • Record finish reason, token counts, revision identity, and trace IDs without logging sensitive prompt content by default.
  • Test malformed role sequences, oversized requests, stop-string edges, Unicode chunk boundaries, cancellations, and partial failures.

Production Decision Checklist

Before rollout:

  • Is the execution identity immutable and visible in traces?
  • Are client and engine timing boundaries documented?
  • Does the load test reproduce prompt/output lengths, arrival rate, burstiness, and cache state?
  • Are p95/p99 TTFT, TPOT/ITL, E2E, goodput, errors, preemptions, and memory within gates?
  • Do task quality, structured outputs, safety, and finish reasons match the baseline?
  • Are tenant isolation, quotas, cancellation, retry, and billing behavior tested?
  • Can the previous engine, model, template, tokenizer, adapter, and scheduler configuration be restored independently?