What is Throughput?
Throughput is the amount of explicitly defined work an LLM serving system completes per unit of wall-clock time across a measured workload, such as successful requests or output Tokens per second.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
Choose the Numerator
Request throughput counts completed requests, output throughput counts generated output Tokens, input throughput counts Prefill Tokens, and total throughput combines input and output under a declared convention. Business throughput can count verified answers, accepted code changes or completed Agent trajectories. Report successful, failed, cancelled and truncated work separately; excluding failures from the numerator without disclosing them overstates capacity.
Find Sustainable Load
Sweep Request Rate or Concurrency over a realistic input/output distribution and identify where the service still meets tail-latency, error, quality and fairness objectives. [DistServe](https://arxiv.org/abs/2401.09670) calls the maximum rate served within TTFT and TPOT objectives Goodput; its reported gains remain specific to its evaluated systems. Closed-loop tests can hide overload because the client waits before sending more work; Open-loop or Trace Replay better represents independent arrivals but must record Schedule Lag. A throughput plateau with rising Queue Depth means the service is saturated even if GPU Utilization looks high.
Aggregate Correctly
Counters such as completed requests and Tokens can be summed across Replicas before division by one aligned window. Percentiles cannot be averaged safely across instances; merge raw samples or compatible Histograms. The [AIPerf metrics reference](https://docs.nvidia.com/aiperf/reference/ai-perf-metrics-reference) explicitly separates request-level records, aggregate counters and derived rates. Pin model, Tokenizer, Runtime, hardware, Replica count, precision, Cache state, arrival distribution, duration and Warmup. Report per-user latency and rate beside aggregate capacity so batching does not hide a degraded interactive experience.
Key Characteristics
- Measures aggregate completed work over one declared benchmark window
- Can use requests, input Tokens, output Tokens, total Tokens or business outcomes
- Changes with offered load, request shape, batching, cache state and replica count
- Can improve while per-user TTFT, TPOT, jitter or fairness worsens
- Reaches a saturation region where queues grow without proportional completion gains
- Becomes Goodput only after success, latency, quality and safety gates are applied
Common Use Cases
- Sizing replicas and accelerators for a production traffic distribution
- Finding the maximum arrival rate that still satisfies latency and quality SLOs
- Comparing schedulers, batching, quantization or parallelism under equal workload
- Calculating cost and energy per accepted task rather than per attempted Token
- Detecting saturation, queue growth, preemption and fairness regressions
Example
Loading code...Frequently Asked Questions
Is LLM Throughput the same as Tokens per Second?
Tokens per Second is one possible throughput unit. A report may instead count successful requests, input Tokens, output Tokens, total Tokens or verified tasks. It must state the numerator, scope and full measurement interval.
What is the difference between Throughput and Goodput?
Throughput counts completed work under the declared rule. Goodput counts only work that also satisfies acceptance criteria such as success, quality, TTFT, TPOT, safety or correctness. High raw throughput can coexist with low Goodput.
Why can higher Throughput increase latency?
Larger or denser batches can amortize model work and raise aggregate capacity, but requests may wait longer or receive Decode service less frequently. The trade-off depends on arrival shape, scheduler, context lengths and hardware.
How do you find sustainable LLM Throughput?
Sweep realistic offered load until latency, error, quality or fairness limits fail, and keep the highest rate that meets every declared SLO. Include warmup, benchmark duration, failures and client schedule lag in the report.
Can throughput be summed across replicas?
Aligned counters such as completed requests or Tokens can be summed over one window. Do not average replica percentiles; merge raw request samples or compatible histogram buckets, and account for uneven routing, failures and replica health.