What is Token?
Token is a model-specific unit represented by an integer ID that a language model consumes or produces; the corresponding text fragment may be a word, subword, byte sequence, punctuation mark, whitespace pattern, or special control symbol.
Quick Facts
| Created | 2010s (modern subword tokenization with BPE) |
|---|---|
| Specification | Official Specification |
How It Works
Token IDs belong to a specific model contract
Tokenizers map text fragments to integer IDs from a fixed vocabulary. The model consumes IDs, not the printed fragments shown by a tokenizer inspector. An ID has meaning only with the matching vocabulary, normalization rules, merge model, and special-token definitions; copying IDs between unrelated models is invalid even when decoded text looks similar.
Count the serialized request, not just visible text
A chat request can add role markers, separators, beginning or end markers, tool schemas, image placeholders, and other control tokens through its chat template. Counting only the user message can therefore understate the actual input. Use the provider's exact tokenizer and template when available, and treat response usage metadata as the authoritative post-request record.
Context capacity is a shared budget
The context limit applies to the complete sequence processed for a generation step: system instructions, conversation history, retrieved evidence, tool definitions or results, template overhead, and generated output. Reserve output capacity and a safety margin before admitting optional context. Do not assume every provider applies the same truncation order or accepts the advertised maximum under every feature combination.
Token counts affect cost and serving behavior
Input and output tokens may be billed or cached differently, and prompt length affects prefill work while generated length affects decoding work. Tokens-per-second is not comparable without separating these phases and stating concurrency and hardware. Character-to-token or word-to-token ratios are rough corpus-dependent estimates only; multilingual text, code, identifiers, numbers, and whitespace can diverge sharply.
Key Characteristics
- Identified by an integer in a tokenizer vocabulary rather than by a universal word boundary
- May represent text fragments, bytes, punctuation, whitespace, or control symbols
- Depends on the exact tokenizer, vocabulary revision, and serialization template
- Shares the model context budget with prompt overhead and generated output
- Influences API usage, prefill work, decode length, cache behavior, and storage
- Cannot be estimated reliably from one fixed words-or-characters conversion ratio
Common Use Cases
- Rejecting or trimming requests before they exceed a model context limit
- Reserving output capacity while assembling prompts, retrieval evidence, and tool schemas
- Reconciling local estimates with provider-reported input, cached, reasoning, and output usage
- Choosing RAG chunk and overlap sizes with the deployed model's actual tokenizer
- Comparing multilingual, code, and structured prompts under identical quality requirements
Example
Loading code...Frequently Asked Questions
Is one token equal to one word or a fixed number of characters?
No. A token boundary depends on the tokenizer vocabulary and input bytes. A common word may be one token in one model and several in another; code, numbers, whitespace, emoji, and different writing systems behave differently. Character or word ratios are useful only for rough capacity planning and must not replace exact encoding.
Why does an API report more input tokens than my visible prompt contains?
The serialized request may include system instructions, role markers, separators, special tokens, tool or response schemas, image placeholders, and conversation history. Local counting must reproduce the provider's tokenizer and chat template. When local and server counts disagree, provider usage metadata is the billing and enforcement record.
Do input tokens plus maximum output always equal the context limit?
They form the core budget, but the exact rule is provider and model specific. Template overhead, hidden control tokens, reasoning-token accounting, multimodal inputs, and feature-specific limits can matter. Reserve a safety margin, test the deployed endpoint, and do not assume that another model's context formula or truncation behavior applies.
Can I reuse token IDs across models that use BPE?
No. BPE describes a family of merge procedures, not a shared vocabulary. Models can use different base symbols, normalization, merge ranks, special tokens, or vocabulary revisions. Token ID 42 in one vocabulary has no guaranteed relationship to ID 42 in another, so IDs and cached token sequences must remain tied to the exact model contract.
How should token usage be measured in production?
Preflight with the exact tokenizer and serialized chat template, record admitted input and reserved output, and compare the estimate with response usage fields. Separate uncached input, cached input, generated output, and any provider-specific reasoning accounting when available. Alert on drift after model, tokenizer, template, or tool-schema changes.