What is Test-Time Compute?
Test-Time Compute is the explicit allocation of additional inference computation to a problem through longer trajectories, multiple candidate samples, search over partial solutions, verification, or iterative refinement without updating the base model weights.
Quick Facts
| Full Name | Test-Time Compute Scaling |
|---|---|
| Created | 2024 (OpenAI o1 series) |
| Specification | Official Specification |
How It Works
Three Inference Regimes
Single-trajectory scaling lets one response continue, revise, or use more internal computation. Leaf-level scaling generates complete candidates and reduces them with majority vote, a reward model, a deterministic checker, or expected utility. Prefix-level scaling expands and prunes partial states with beam, best-first, or tree-search policies.
These regimes have different latency and cost profiles. Parallel samples can reduce wall-clock latency relative to sequential search while consuming similar aggregate compute; a longer trace can still repeat one misconception; prefix search adds evaluator calls and coordination overhead.
Generator, Verifier, and Reducer
Scaling requires both candidate discovery and a rule for selecting or aggregating candidates. Deterministic tests, formal checkers, or environment outcomes can provide strong evidence in bounded domains. Majority vote only measures agreement, and a learned verifier can share the generator's bias or reward polished but invalid reasoning.
Coverage, the chance that at least one generated candidate is correct, is not the same as selected-answer accuracy. Without a reliable reducer, generating more candidates may enlarge the pool without helping the system identify the right result.
Budget Allocation and Stopping
A production policy states the budget in measurable units such as model invocations, generated and verifier tokens, FLOPs, tool calls, wall-clock deadline, or monetary cost. It also defines whether work is sequential or parallel and what evidence allows early stopping.
Uniformly assigning a large budget wastes resources on easy tasks and can amplify errors on tasks outside the verifier's competence. Difficulty estimates, pilot-sample disagreement, verifier margin, remaining deadline, and expected value can guide allocation, but each signal needs calibration on the target workload.
Evaluation and Failure Modes
Compare strategies at matched end-to-end compute and include the base model, prompt, sampler, answer normalizer, verifier, stopping rule, hardware, concurrency, and failure handling in the release identity. Report accuracy or utility together with latency percentiles, total compute, cost, abstention, and unresolved failures.
Extra inference can fail through correlated samples, weak or exploitable verifiers, answer-normalization errors, search collapse, premature stopping, timeouts, and unverifiable open-ended tasks. High-stakes decisions still need independent evidence and deterministic controls; longer reasoning is not a safety guarantee.
Key Characteristics
- Inference-scoped: spends additional compute per problem without changing the base checkpoint
- Multi-regime: includes longer trajectories, complete-candidate sampling, and partial-state search
- Verifier-dependent: gains become actionable only when candidates can be compared reliably
- Budgeted: calls, tokens, FLOPs, tools, latency, and money are explicit resources
- Difficulty-sensitive: the best strategy and budget vary with task and base-model capability
- System-evaluated: selection accuracy, latency, cost, failures, and stopping matter beyond candidate coverage
Common Use Cases
- Sampling several code patches and selecting only one that passes isolated deterministic tests
- Generating multiple mathematical solutions and using a sound answer checker before selection
- Expanding partial plans while a calibrated verifier prunes low-value branches under a deadline
- Routing easy requests to a single attempt and difficult qualified requests to a larger inference budget
- Comparing a smaller model with scaled inference against a larger model at matched quality and total cost
Example
Loading code...Frequently Asked Questions
Is ordinary autoregressive generation a single forward pass?
No. A decoder-only language model normally performs a sequence of forward steps, usually one generated token at a time while reusing a KV cache. Test-time scaling adds computation beyond a chosen baseline through longer trajectories, more candidates, search, verification, tools, or revision.
Does more test-time compute always improve accuracy?
No. Benefits depend on the task, base model, proposal diversity, verifier, and budget. More compute can repeat correlated mistakes, favor an exploitable score, or waste resources on an unverifiable task. Measure a quality-cost curve and stop when marginal value no longer justifies the additional budget.
What is the difference between self-consistency and best-of-N?
Self-consistency samples multiple reasoning paths and aggregates normalized final answers, commonly by frequency. Best-of-N scores complete candidates and selects the highest-scoring one. Agreement is itself a weak verifier; a separate checker can use stronger task evidence, but its errors and cost must be included.
How should test-time compute cost be measured?
Report the complete system budget: generator and verifier invocations, input, output, and reasoning tokens when exposed, FLOPs if available, tool work, concurrency, latency percentiles, failures, and monetary cost. A token multiplier from one model or task is not a portable cost rule.
Are reasoning models the same as test-time compute?
No. A reasoning model may expose controls that spend more internal inference compute, but test-time compute is broader and can wrap any suitable model with sampling, search, external verification, or iterative refinement. The model, inference policy, verifier, and stopping rule are separate parts of the system.