What is Temperature?
Temperature is a decoding parameter that divides next-token logits before softmax, sharpening the probability distribution when positive values are lower and flattening it when values are higher.
Quick Facts
| Full Name | Large Language Model Sampling Temperature |
|---|---|
| Created | Concept originated from statistical mechanics, applied to NLP in 2010s |
| Specification | Official Specification |
How It Works
Temperature rescales relative logit gaps
Subtracting a common value from logits does not change softmax probabilities, so implementations usually subtract the maximum for numerical stability before dividing by T. Lower positive T magnifies logit differences and higher T compresses them. The ranking of finite logits remains unchanged; what changes is probability mass and therefore the chance that sampling selects each candidate.
Zero is an API convention, not the formula
The softmax expression is undefined at T=0. Providers may interpret zero as greedy decoding, clamp it to a small positive value, reject it, or expose no temperature control for a model. Greedy token choice also does not guarantee byte-for-byte repeatability: ties, floating-point kernels, batching, model revisions, hidden prompts, tool results, and backend routing can still change output.
Filters and structured decoding alter the candidate set
Top-p, top-k, repetition controls, logit bias, grammar constraints, and safety filters can operate before or after temperature depending on the serving implementation. Constrained decoding may force syntactically valid outputs while temperature redistributes probability only among currently allowed tokens. Because operation order is provider-specific, changing several controls at once makes causal tuning difficult.
Tune against a task distribution, not folklore
Build a representative evaluation set and measure validity, task quality, factual support, refusal behavior, diversity across repeated samples, latency, and cost. Sweep only supported values while holding model version, prompt, seed if available, and other decoding controls fixed. Choose a setting from the quality-diversity frontier; low temperature is not an accuracy guarantee, and high temperature is not a creativity guarantee.
Key Characteristics
- Divides next-token logits before softmax for positive temperature values
- Changes probability concentration without changing the ordering of finite logits
- Uses provider-specific ranges, defaults, validation, and zero-value behavior
- Interacts with top-p, top-k, penalties, logit bias, constraints, and safety filters
- Cannot by itself guarantee determinism, correctness, factuality, or creativity
- Should be selected through repeated task-level evaluation rather than fixed recipes
Common Use Cases
- Measuring the quality-diversity tradeoff across repeated generation samples
- Concentrating a classification or extraction distribution while retaining fallback checks
- Exploring alternative candidates for brainstorming under later ranking or review
- Testing whether a structured decoder remains robust across supported sampling settings
- Recording decoding configuration so model evaluations and incidents are reproducible
Example
Loading code...Frequently Asked Questions
What does temperature mathematically change in an LLM?
For positive T, each next-token logit is divided by T before softmax. T below 1 magnifies gaps, placing more probability on already higher-logit tokens; T above 1 compresses gaps and spreads probability more broadly. It does not reorder finite logits or retrain the model, and it supplies no external evidence.
Does temperature zero make an LLM deterministic?
Not necessarily. The mathematical formula is undefined at zero, so an API usually substitutes a provider-specific greedy or near-greedy path. Equal scores, numerical kernels, dynamic batching, backend changes, model revisions, hidden instructions, and nondeterministic tools can still alter results. Treat reproducibility as a measured system property.
Does a lower temperature reduce hallucinations?
It can make outputs less variable and concentrate them on high-probability continuations, but high probability is not the same as truth. A model can repeat the same unsupported claim very consistently. Factual systems need grounded evidence, citation checks, calibrated abstention, retrieval evaluation, and task-specific tests in addition to decoding controls.
Should temperature and top-p be changed together?
They affect different parts of sampling: temperature reshapes probabilities, while top-p limits candidates by cumulative probability. Their exact operation order can differ by provider. Hold one control fixed while sweeping the other, record all settings, and evaluate repeated samples before deciding whether a combined configuration is justified.
What temperature should I use for code, extraction, or creative writing?
There is no portable value by task label. Models, prompts, constraints, and provider ranges differ. Start from the documented default, create representative cases, sweep supported values, and compare correctness or validity with diversity, latency, and cost. For strict formats, grammar or schema constraints are stronger controls than temperature alone.