Reasoning Effort in Production: Routing and Evaluation

Direct Answer

Reasoning effort is a versioned inference control, not one cross-provider standard and not a window into a model's private algorithm. Some APIs expose discrete effort levels, some expose adaptive thinking, some accept a soft token budget, and some do not allow thinking to be disabled. Parameter names, defaults, accounting, output visibility, tool behavior, and supported values can all vary by model snapshot.

Production teams should treat effort as an experiment variable. Freeze the task and answer contract, compare supported settings, reserve enough answer space, validate external outcomes, and route only when the router beats simple always-low and always-high baselines.

Four Contracts Hidden Behind “Thinking Mode”

1. Control contract

The request might expose a level such as low or high, a mode such as adaptive, or a token-like budget. The control may be soft guidance rather than a hard compute limit. Unsupported values can fail the request.

2. Accounting contract

Reasoning work may count toward generated-token limits, the context window, and billing even when it is not visible. A request can exhaust its output budget before emitting a complete answer. Cost analysis therefore needs total generated work, incomplete responses, retries, and accepted outcomes.

3. Visibility contract

The API may return no reasoning, a summary, an omitted or encrypted block, or model-generated chain-of-thought text. These artifacts are not equivalent. They should not be stored by default, and they are not reliable evidence that a particular internal search, verification, or policy check occurred.

4. Conversation and tool contract

Providers differ in how reasoning state is carried between turns, whether signatures must be round-tripped, how prompt caching behaves across modes, and whether thinking can occur between tool calls. An adapter must preserve the provider contract without exposing it as a universal application invariant.

Current Provider Patterns

The table describes interface families, not permanent product specifications.

Always read the exact model page before deployment. Pin model ID, API revision, SDK version, request shape, response fields, usage fields, cache behavior, and tested tool sequence.

What Extra Reasoning Can and Cannot Do

Extra inference work can help a model explore alternatives, plan tools, or solve multi-step problems under some protocols. It does not establish:

• correctness or factual support;
• a faithful chain of causation;
• permission to access data or execute a tool;
• adequate test coverage;
• a fixed quality gain;
• a portable latency or cost multiplier.

The DART paper studies routing between direct and extended reasoning under specific math and coding benchmarks. It also documents an important measurement artifact: a one-phase comparison can make a reasoning path look worse simply because the shared generation cap truncates the final answer. That finding supports budget-aware test-time compute experiments; it does not make draft agreement a universal production router.

Build a Matched Evaluation

Freeze the task unit

Each case should include:

• input and authorized evidence;
• expected result or executable verifier;
• allowed refusal and escalation behavior;
• tool and side-effect policy;
• maximum answer and wall-clock budget;
• business impact and slice labels.

Do not use one “difficulty” label as ground truth. A short question can require fresh evidence, while a long question can have a deterministic transformation.

Compare complete strategies

At minimum run:

1. the lowest supported effort;
2. the default effort;
3. a higher effort;
4. a simple policy router;
5. any learned or multi-call router only after the first four.

Give each strategy enough room to produce a final answer, or record truncation as a distinct outcome. Match tool access, retrieval, prompt, timeout, and evaluator. If a higher effort uses a different model snapshot, report it as a model-plus-effort change.

Measure what the user receives

Report results by task slice. An average can hide a router that saves cost on easy tasks while sending a high-impact slice to an inadequate mode.

A Safe Routing Architecture

The router should use signals available before or during execution that have been validated on the target workload. Prompt length, keywords, model self-confidence, and visible reasoning length are weak proxies unless calibrated.

Use a single bounded escalation rather than an unconstrained loop. After an ambiguous timeout, reconcile tool state before retrying. Never replay a payment, deployment, message, or deletion because a high-effort response failed to arrive.

Provider Adapter Requirements

A provider-neutral request can describe intent:

quality tier: economical | balanced | thorough
hard output limit
deadline
tool policy
required evidence
fallback policy

The adapter maps that intent to a provider's supported API. It must reject unsupported mappings instead of silently substituting a different mode.

Record in every trace:

• provider, endpoint, model snapshot, SDK and API revision;
• requested abstract tier and concrete mapped parameters;
• input, cached, output, and reasoning usage when available;
• stop reason, incomplete details, tool calls, retries, and latency;
• answer verifier result and final route.

Do not log raw secrets, unrestricted prompts, private tool results, or reasoning blocks without a documented purpose and retention policy.

Runnable Go Router

This standard-library example chooses among measured modes for one task class. It first enforces quality, incomplete-response, and latency gates, then minimizes cost per verified task. The records are synthetic fixtures.

package main

import (
	"fmt"
	"sort"
)

type ModeResult struct {
	Name                string
	TaskSuccess         float64
	FalseAcceptRate     float64
	IncompleteRate      float64
	P95LatencyMS        int
	CostPerVerifiedTask float64
}

type Gate struct {
	MinTaskSuccess     float64
	MaxFalseAcceptRate float64
	MaxIncompleteRate  float64
	MaxP95LatencyMS    int
}

func passes(result ModeResult, gate Gate) bool {
	return result.TaskSuccess >= gate.MinTaskSuccess &&
		result.FalseAcceptRate <= gate.MaxFalseAcceptRate &&
		result.IncompleteRate <= gate.MaxIncompleteRate &&
		result.P95LatencyMS <= gate.MaxP95LatencyMS
}

func main() {
	gate := Gate{
		MinTaskSuccess:     0.88,
		MaxFalseAcceptRate: 0.01,
		MaxIncompleteRate:  0.02,
		MaxP95LatencyMS:    2500,
	}
	results := []ModeResult{
		{Name: "low", TaskSuccess: 0.84, FalseAcceptRate: 0.008, IncompleteRate: 0.002, P95LatencyMS: 650, CostPerVerifiedTask: 0.010},
		{Name: "medium", TaskSuccess: 0.90, FalseAcceptRate: 0.007, IncompleteRate: 0.006, P95LatencyMS: 1300, CostPerVerifiedTask: 0.017},
		{Name: "high", TaskSuccess: 0.92, FalseAcceptRate: 0.006, IncompleteRate: 0.031, P95LatencyMS: 3100, CostPerVerifiedTask: 0.034},
	}

	eligible := make([]ModeResult, 0, len(results))
	for _, result := range results {
		if passes(result, gate) {
			eligible = append(eligible, result)
		}
	}
	if len(eligible) == 0 {
		fmt.Println("route=review")
		return
	}

	sort.Slice(eligible, func(i, j int) bool {
		return eligible[i].CostPerVerifiedTask < eligible[j].CostPerVerifiedTask
	})
	fmt.Printf("route=%s cost_per_verified_task=%.3f\n",
		eligible[0].Name,
		eligible[0].CostPerVerifiedTask,
	)
}

Expected output:

route=medium cost_per_verified_task=0.017

The selected mode is valid only for the fixture's task class and policy. Re-run the experiment after changing the model, prompt, tool set, retrieval, output limit, or evaluator.

Common Failure Patterns

Treating high effort as a quality flag

Higher effort may improve one slice and degrade another. Keep the lower-effort baseline and use paired case-level analysis to find which failures actually changed.

Sharing one cap between reasoning and answer without inspection

A request can spend its allowance before producing the required answer. Monitor incomplete reasons and reserve answer space according to the provider contract.

Using a second model as unquestioned verifier

An LLM-as-a-judge can share the candidate's bias, prefer verbosity, or be manipulated by untrusted content. Calibrate it against human labels, randomize order for pairwise tests, and retain deterministic or executable checks.

Routing by surface form

Rules such as “math symbols mean high effort” or “short prompts mean low effort” miss hard factual, safety, and tool-use cases. Route by evaluated task contracts and business risk.

Storing reasoning as privileged evidence

Visible rationales may be incomplete or unfaithful. Anthropic's reasoning faithfulness research shows that models can omit influential hints from their stated reasoning under the studied conditions. Audit inputs, sources, tool calls, policy checks, and effects instead.

Frequently Asked Questions

Is reasoning effort the same as temperature?

No. Effort controls provider-specific inference work; temperature affects sampling on APIs that expose it. Some reasoning model versions restrict sampling parameters. Check the exact model contract rather than assuming the controls are independent or supported.

Can thinking be disabled?

It depends on the model. Some interfaces allow a disabled or minimal mode, while others require reasoning and only expose lower effort. Unsupported combinations can return an error.

Are reasoning tokens part of the context window?

Often they count toward generated output and the model's context accounting, but exact semantics are provider-specific. Inspect usage and incomplete-response fields for the deployed model.

Should a router use model confidence?

Only after calibration against labeled outcomes for the same task and version. Verbal confidence and token probabilities are not automatically calibrated correctness probabilities.

What is the safest default?

Use the provider's supported default only as an experimental starting point. The production default is the least costly mode that passes the task's quality, risk, latency, and recovery gates.

Primary Sources

• OpenAI reasoning model guide
• Anthropic adaptive thinking documentation
• Google Cloud Gemini thinking documentation
• DART: adaptive thinking-budget routing
• Reasoning models do not always say what they think
• Test-time compute engineering