What is Tokens per Second?
Tokens per Second is a rate that must be qualified by token type, scope and timing interval, such as per-request output Tokens during Decode or aggregate output Tokens across a serving system.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
Token and Event Identity
Use the target model's Tokenizer or authoritative server usage fields; characters, words and streamed Chunks are not Tokens. The [AIPerf metrics reference](https://docs.nvidia.com/aiperf/reference/ai-perf-metrics-reference) separates output sequence length, reasoning, Cache and speculative-token fields rather than treating every transport event alike. A Chunk may carry zero, one or several Tokens. State whether the first output Token is excluded, whether the interval starts at request send or first output, and whether failed, cancelled or truncated requests enter the result.
Per-Request Versus Aggregate Rate
Per-request output TPS describes a user's Decode cadence. Aggregate output TPS describes service capacity under concurrent load. Larger batches can raise aggregate TPS while lowering each stream's rate or increasing jitter. Input TPS measures Prefill work and is not directly comparable with output TPS because the phases have different computation, batching and state behavior. Total-token TPS can be useful for one fixed workload but can hide a shift between input and output.
Comparison Contract
A comparable result fixes model and Tokenizer revisions, actual input/output length distributions, concurrency, arrival process, Cache state, precision, sampling and stop policy, Runtime, hardware, region, Warmup and Benchmark Duration. The [vLLM benchmark CLI](https://docs.vllm.ai/en/latest/benchmarking/cli/) reports Request, Output-Token and Total-Token Throughput separately from TTFT, TPOT and ITL. Report those latency distributions beside per-user and aggregate TPS, plus error rate and task quality. A high rate from shorter outputs, early termination or an easier Tokenizer is not a serving improvement.
Key Characteristics
- Requires a declared numerator: input, output, accepted or total Tokens
- Requires a declared scope: request, user stream, accelerator or service
- Requires a declared interval: Decode Duration, end-to-end time or benchmark window
- Depends on Tokenizer, model, output policy, batching, load, cache and hardware
- Can move opposite to TTFT, tail latency or per-user fairness
- Has value only when quality, errors and workload identity are preserved
Common Use Cases
- Measuring visible generation cadence for one streaming request
- Estimating aggregate serving capacity under realistic concurrency
- Comparing runtimes only after aligning Token and timing definitions
- Detecting Decode regressions after quantization, batching or scheduler changes
- Calculating cost or energy per accepted output Token without hiding failed work
Example
Loading code...Frequently Asked Questions
Does Tokens per Second mean per-user speed or total server throughput?
It can mean either, which is why an unqualified TPS value is ambiguous. State whether the numerator is input, output or total Tokens; whether the scope is one stream or the whole service; and which wall-clock interval forms the denominator.
Should the first output Token be included in Decode TPS?
For post-first-token per-request TPS, exclude it and divide `output_tokens - 1` by the interval from first to last output. Aggregate benchmark tools may use a different formula, so preserve the tool version and metric definition instead of silently mixing results.
Why is a streamed Chunk not a Token?
Streaming is a transport behavior. One Chunk can contain no visible text, one Token or several Tokens, especially with buffering or speculative decoding. Count with the model Tokenizer or trusted usage fields, and use Chunk timing separately when measuring delivery jitter.
Can higher aggregate TPS make each user's stream slower?
Yes. More batching can amortize model-weight reads and increase total output rate while each sequence is scheduled less often. Report per-user TPOT or TPS and tail latency beside aggregate throughput.
Can TPS compare different models or providers fairly?
Only under a controlled task and workload. Different Tokenizers, output lengths, stop behavior, quality, regions and serving policies change the count and interval. Compare successful-task quality and completion time as well as Token rate.