Direct Answer

AI Agent model routing is a constrained decision system that chooses the lowest-cost execution path that still satisfies a step's quality, latency, and safety contract. A production policy must account for router and verifier overhead, failed first attempts, fallbacks, tool side effects, and end-to-end task success. The right objective is therefore not the cheapest model or token, but the lowest cost per accepted Agent trajectory under explicit rollback thresholds.

This page focuses on routing decisions inside an AI Agent. The broader AI inference cost economics guide covers the separate accounting problem of comparing hosted APIs, private runtimes, and edge deployments.

Key Takeaways

  • Route at the Agent step level when model requirements change across planning, retrieval, tool use, and response synthesis.
  • A router decides before generation; a cascade evaluates an output and decides whether to stop, retry, escalate, abstain, or request approval.
  • Optimize cost per accepted trajectory, not provider price, parameter count, or average benchmark score.
  • False acceptance is usually more dangerous than false escalation because an invalid tool call can create irreversible side effects.
  • Evaluate with historical replay, shadow traffic, bounded canaries, and automatic rollback. Offline benchmark wins are not deployment evidence.
  • Treat the route policy, prompts, model endpoints, validators, and evaluation dataset as one versioned release unit.

What Problem Does Agent Model Routing Solve?

Agent model routing solves a workload-allocation problem: different steps in one trajectory can have different capability and risk requirements. A planning step may need broad reasoning, while a schema-constrained lookup may need reliable argument construction rather than open-ended prose. Sending every step to one model can waste money, but sending every "simple-looking" step to a smaller model can silently damage task success.

This creates three distinct decisions:

Decision Evidence available Typical action Main risk
Static assignment Step type known at design time Bind a route to a workflow node Workload drift makes the assignment stale
Pre-generation routing Request, state, risk, and model metadata Select one candidate before inference Router predicts the wrong capability need
Output-aware cascade Initial output plus validator signals Accept, retry, escalate, abstain, or approve First-pass work and verifier cost are wasted

Research such as RouteLLM and FrugalGPT shows that learned routers and cascades can trade quality against cost on their evaluated model pools and datasets. Those results establish useful mechanisms, not universal savings. LLMRouterBench further reports that routing performance depends strongly on model-pool construction and that sophisticated routers do not always beat simple baselines.

Agent execution adds another boundary. A chat answer can be judged after generation, but a tool call may mutate production state. Query-level routing results therefore do not automatically transfer to conditional, step-level Agent execution. TwinRouterBench and Switchcraft examine this more specific setting, including per-call routing and tool-call correctness.

LLMRouter and xRouteBench add a broader warning: router rankings can change across task families and cost budgets, and extra multi-turn routing does not consistently beat a single-turn policy. Treat the candidate pool, workload, user context, history available at each decision, task metric, and cost ledger as part of the evaluated system. A router result is not portable when those inputs change.

Router vs Cascade: Choose the Decision Point

A router is best when request features predict the required capability before generation; a cascade is best when output evidence materially improves the decision. Many production systems combine both.

flowchart LR A["Agent step and state"] --> B["Risk and policy checks"] B --> C["Pre-generation router"] C --> D["Candidate model"] D --> E["Typed output and evidence"] E --> F{"Quality gate passes?"} F -->|Yes| G["Commit step result"] F -->|Retryable| H["Retry or fallback"] F -->|Uncertain| I["Escalate or abstain"] F -->|High risk| J["Human approval"] H --> E I --> E

Use the following decision rule:

Condition Prefer Why
Stable step taxonomy and low route ambiguity Static or rule-based route Easy to audit and cheap to operate
Strong request features predict model suitability Pre-generation router Avoids paying for a failed first attempt
Correctness can be checked after generation Cascade Output-aware validation improves stop decisions
Side effects are irreversible or regulated Approval or abstention gate A fallback cannot undo an unsafe commit
No candidate meets the contract Abstain Forced routing hides unsupported work

A confidence score is not automatically a correctness probability. Before using it as an escalation threshold, test calibration by route, task family, language, tool, and risk tier. Provider or prompt changes can invalidate that calibration even when the router code is unchanged.

Define the Route Contract Before Choosing Models

A route contract states what must remain true regardless of which model executes the step. Without it, "use a cheaper model" is not an engineering decision because there is no accepted-output boundary.

Define at least these fields:

yaml
step_contract:
  step_type: "create_refund"
  risk_tier: "high"
  allowed_tools: ["payments.get_order", "payments.create_refund"]
  output_schema: "refund_decision_v3"
  max_end_to_end_ms: 3500
  max_attempts: 2
  requires_evidence: true
  requires_human_approval: true
  abstain_when:
    - "order ownership is uncertain"
    - "requested amount exceeds policy"

The contract should cover:

  1. Task boundary: what the step may decide and what belongs to another component.
  2. Typed output: schema, required evidence, and allowed tool names.
  3. Authorization: identity, tenant, data scope, and action permissions.
  4. Quality gate: deterministic checks, task-specific evaluator, or human review.
  5. Latency budget: total route time, including router, verifier, retries, and fallback.
  6. Failure policy: retry, alternate model, alternate tool, abstain, or approval.
  7. Side-effect boundary: which actions can be simulated and which require a commit gate.

The surrounding Agent harness should enforce these constraints outside the model. The model may propose an action, but policy code should own authorization and irreversible commits.

Route by Capability, Risk, and State

Good route features describe the work and its consequences rather than assuming a model class is always sufficient.

Capability Features

  • Step type: planning, extraction, ranking, synthesis, code edit, or tool call
  • Input language and modality
  • Context length and evidence density
  • Required output schema and tool catalog size
  • Need for long-horizon state or cross-step consistency
  • Historical acceptance rate for the candidate on the same slice

Risk Features

  • Read-only versus state-changing operation
  • Reversibility and monetary impact
  • Personal, confidential, or regulated data
  • External communication or code execution
  • Required approval and audit evidence

Runtime Features

  • Current endpoint latency and error rate
  • Token and context budget remaining
  • Prior failures in the trajectory
  • Provider availability and regional constraints
  • Route-specific queue depth and private-runtime capacity

Do not route solely on prompt length, model parameter count, or a generic "complexity" label. These proxies can correlate with difficulty on one dataset and fail on another. The route decision should use features that can be logged, reproduced, and evaluated against accepted outcomes.

Measure Accepted Steps and Accepted Trajectories

A route is successful only when the step and the overall trajectory satisfy their contracts. Local validity is necessary but insufficient: valid JSON can contain the wrong tool, valid arguments can target the wrong account, and individually plausible steps can still fail the user task.

Track both levels:

Level Required evidence
Step acceptance Schema valid, correct tool and arguments, evidence grounded, policy allowed, side effect verified
Trajectory acceptance User goal achieved, no unsafe action, final state consistent, latency SLO met, no unhandled recovery

Core route metrics include:

text
step_acceptance_rate =
  accepted_steps / attempted_steps

trajectory_acceptance_rate =
  accepted_trajectories / started_trajectories

false_acceptance_rate =
  invalid_outputs_accepted_by_gate / outputs_accepted_by_gate

escalation_rate =
  escalated_steps / routed_steps

wasted_first_pass_cost =
  cost_of_rejected_initial_attempts

cost_per_accepted_trajectory =
  total_route_cost / accepted_trajectories

total_route_cost must include the router, every model attempt, validators, fallbacks, tool execution, human review, and correction work. For private runtimes, allocate infrastructure and operations using the same boundary as the general inference-cost model. Otherwise an API route and a self-hosted route are not comparable.

Expected Cascade Cost

For a two-stage cascade, the minimum useful expectation is:

text
expected_route_cost =
    initial_route_cost
  + p_escalate * fallback_cost
  + verifier_cost
  + p_false_accept * correction_and_risk_cost

The last term prevents a cheap but unsafe gate from looking efficient. If the consequence of false acceptance cannot be credibly priced, treat it as a hard constraint rather than assigning an arbitrary dollar value.

Build a Reproducible Route-Policy Evaluation

The following dependency-free Go program compares candidate policies on recorded route outcomes. It does not call a model. Instead, it evaluates evidence produced by replay or shadow runs, rejects duplicate trajectory IDs, and requires every policy to cover the same trajectory set. This pairing prevents an easier sample mix from masquerading as a better route policy.

Save this as evaluate_routes.go:

go
package main

import (
	"bufio"
	"bytes"
	"encoding/json"
	"errors"
	"flag"
	"fmt"
	"io"
	"math"
	"os"
	"sort"
)

type Outcome struct {
	Policy       string  `json:"policy"`
	TrajectoryID string  `json:"trajectory_id"`
	Accepted     bool    `json:"accepted"`
	GateAccepted bool    `json:"gate_accepted"`
	Escalated    bool    `json:"escalated"`
	LatencyMS    float64 `json:"latency_ms"`
	RouteCost    float64 `json:"route_cost"`
}

type PolicyReport struct {
	Trajectories              int      `json:"trajectories"`
	AcceptanceRate            float64  `json:"acceptance_rate"`
	FalseAcceptanceRate       float64  `json:"false_acceptance_rate"`
	EscalationRate            float64  `json:"escalation_rate"`
	P95LatencyMS              float64  `json:"p95_latency_ms"`
	LatencySLOPassRate        float64  `json:"latency_slo_pass_rate"`
	TotalCost                 float64  `json:"total_cost"`
	CostPerAcceptedTrajectory *float64 `json:"cost_per_accepted_trajectory"`
}

func decodeOutcome(line []byte) (Outcome, error) {
	var outcome Outcome
	decoder := json.NewDecoder(bytes.NewReader(line))
	decoder.DisallowUnknownFields()
	if err := decoder.Decode(&outcome); err != nil {
		return Outcome{}, err
	}
	if err := decoder.Decode(&struct{}{}); !errors.Is(err, io.EOF) {
		return Outcome{}, errors.New("expected one JSON object")
	}
	if outcome.Policy == "" || outcome.TrajectoryID == "" {
		return Outcome{}, errors.New("policy and trajectory_id must be non-empty")
	}
	if outcome.LatencyMS < 0 || outcome.RouteCost < 0 {
		return Outcome{}, errors.New("latency_ms and route_cost must be nonnegative")
	}
	return outcome, nil
}

func readOutcomes(path string) ([]Outcome, error) {
	file, err := os.Open(path)
	if err != nil {
		return nil, err
	}
	defer file.Close()

	var outcomes []Outcome
	scanner := bufio.NewScanner(file)
	scanner.Buffer(make([]byte, 64*1024), 1024*1024)
	for lineNumber := 1; scanner.Scan(); lineNumber++ {
		line := bytes.TrimSpace(scanner.Bytes())
		if len(line) == 0 {
			continue
		}
		outcome, err := decodeOutcome(line)
		if err != nil {
			return nil, fmt.Errorf("%s:%d: %w", path, lineNumber, err)
		}
		outcomes = append(outcomes, outcome)
	}
	if err := scanner.Err(); err != nil {
		return nil, err
	}
	if len(outcomes) == 0 {
		return nil, errors.New("input contains no outcomes")
	}
	return outcomes, nil
}

func percentile(values []float64, quantile float64) float64 {
	ordered := append([]float64(nil), values...)
	sort.Float64s(ordered)
	index := int(math.Ceil(float64(len(ordered))*quantile)) - 1
	if index < 0 {
		index = 0
	}
	return ordered[index]
}

func evaluate(outcomes []Outcome, latencySLOMS float64) (map[string]PolicyReport, error) {
	grouped := make(map[string][]Outcome)
	idsByPolicy := make(map[string]map[string]struct{})
	for _, outcome := range outcomes {
		if idsByPolicy[outcome.Policy] == nil {
			idsByPolicy[outcome.Policy] = make(map[string]struct{})
		}
		if _, exists := idsByPolicy[outcome.Policy][outcome.TrajectoryID]; exists {
			return nil, fmt.Errorf(
				"duplicate trajectory %q for policy %q",
				outcome.TrajectoryID,
				outcome.Policy,
			)
		}
		idsByPolicy[outcome.Policy][outcome.TrajectoryID] = struct{}{}
		grouped[outcome.Policy] = append(grouped[outcome.Policy], outcome)
	}
	if len(grouped) < 2 {
		return nil, errors.New("at least two policies are required")
	}

	var reference map[string]struct{}
	for policy, ids := range idsByPolicy {
		if reference == nil {
			reference = ids
			continue
		}
		if len(ids) != len(reference) {
			return nil, fmt.Errorf("policy %q has an unpaired trajectory set", policy)
		}
		for id := range reference {
			if _, exists := ids[id]; !exists {
				return nil, fmt.Errorf("policy %q is missing trajectory %q", policy, id)
			}
		}
	}

	report := make(map[string]PolicyReport, len(grouped))
	for policy, rows := range grouped {
		accepted, gateAccepted, falseAccepts, escalated := 0, 0, 0, 0
		totalCost := 0.0
		latencies := make([]float64, 0, len(rows))
		latencyPasses := 0
		for _, row := range rows {
			if row.Accepted {
				accepted++
			}
			if row.GateAccepted {
				gateAccepted++
				if !row.Accepted {
					falseAccepts++
				}
			}
			if row.Escalated {
				escalated++
			}
			if row.LatencyMS <= latencySLOMS {
				latencyPasses++
			}
			totalCost += row.RouteCost
			latencies = append(latencies, row.LatencyMS)
		}

		falseAcceptanceRate := 0.0
		if gateAccepted > 0 {
			falseAcceptanceRate = float64(falseAccepts) / float64(gateAccepted)
		}
		var costPerAccepted *float64
		if accepted > 0 {
			value := totalCost / float64(accepted)
			costPerAccepted = &value
		}
		report[policy] = PolicyReport{
			Trajectories:              len(rows),
			AcceptanceRate:            float64(accepted) / float64(len(rows)),
			FalseAcceptanceRate:       falseAcceptanceRate,
			EscalationRate:            float64(escalated) / float64(len(rows)),
			P95LatencyMS:              percentile(latencies, 0.95),
			LatencySLOPassRate:        float64(latencyPasses) / float64(len(rows)),
			TotalCost:                 totalCost,
			CostPerAcceptedTrajectory: costPerAccepted,
		}
	}
	return report, nil
}

func main() {
	latencySLOMS := flag.Float64("latency-slo-ms", 0, "positive latency SLO")
	flag.Parse()
	if *latencySLOMS <= 0 || flag.NArg() != 1 {
		fmt.Fprintln(os.Stderr, "usage: go run evaluate_routes.go --latency-slo-ms 3000 outcomes.jsonl")
		os.Exit(2)
	}

	outcomes, err := readOutcomes(flag.Arg(0))
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	report, err := evaluate(outcomes, *latencySLOMS)
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	encoder := json.NewEncoder(os.Stdout)
	encoder.SetIndent("", "  ")
	if err := encoder.Encode(report); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
}

Each JSONL row represents one completed trajectory under one policy. Every compared policy must include the same trajectory IDs:

json
{"policy":"baseline","trajectory_id":"t-001","accepted":true,"gate_accepted":true,"escalated":false,"latency_ms":1800,"route_cost":0.042}
{"policy":"baseline","trajectory_id":"t-002","accepted":true,"gate_accepted":true,"escalated":false,"latency_ms":1500,"route_cost":0.036}
{"policy":"candidate","trajectory_id":"t-001","accepted":true,"gate_accepted":true,"escalated":true,"latency_ms":2400,"route_cost":0.031}
{"policy":"candidate","trajectory_id":"t-002","accepted":false,"gate_accepted":true,"escalated":false,"latency_ms":900,"route_cost":0.008}

Run it with:

bash
go run evaluate_routes.go --latency-slo-ms 3000 outcomes.jsonl

Do not approve a policy from the aggregate alone. Slice the input by task family, language, risk tier, tool, provider, and route decision. A policy can improve the average while regressing a small high-impact cohort.

Evaluate Before Production Traffic

A safe rollout separates policy evaluation from user impact.

1. Historical Replay

Replay a frozen, versioned evaluation set through every candidate path. Include ordinary requests, previously failed trajectories, tool errors, authorization denials, ambiguous inputs, and adversarial cases. Prevent future information from leaking into historical state.

2. Shadow Evaluation

Run the candidate policy beside the production route without committing its actions. Compare route choices, accepted outputs, cost, and latency. For state-changing tools, use simulators or read-only validation so the shadow path cannot duplicate side effects.

3. Bounded Canary

Expose a small, explicitly eligible segment. Exclude high-risk actions until their approval path has separate evidence. Keep the old route available and use deterministic assignment so comparisons are reproducible.

4. Automatic Rollback

Rollback should trigger on guardrails, not on cost alone:

  • trajectory acceptance falls below its threshold;
  • false acceptance exceeds its threshold;
  • unauthorized or duplicate side effects occur;
  • p95 or p99 end-to-end latency breaches the SLO;
  • escalation or abstention rate shifts beyond the expected band;
  • route distribution collapses to one candidate;
  • model, provider, prompt, or validator version changes without evaluation.

Agent observability should connect each route decision to the final trajectory outcome. Log the policy version, features used, candidate set, selected route, gate result, escalation reason, costs, latency, and side-effect receipt. Do not log sensitive prompts or credentials merely to make routing easier to debug.

Where Small Language Models Fit

A Small Language Model is one candidate class, not a routing policy. It may be effective for repetitive, bounded steps when evaluation demonstrates acceptable tool and trajectory outcomes. The NVIDIA Research SLM-for-Agents position paper argues for heterogeneous Agent systems, but it is a position paper rather than proof that an SLM is universally cheaper or more reliable.

Use the same evidence requirements for every candidate:

  • Can it produce the required typed output?
  • Does it choose the correct tool and arguments on the target distribution?
  • Does quantization or runtime configuration change acceptance?
  • What are its retry, escalation, and abstention rates?
  • Does it fit the latency and infrastructure boundary?
  • What is its cost per accepted trajectory after all fallback work?

Parameter count does not answer these questions. Neither does a generic reasoning benchmark. Model selection remains a workload-specific evaluation, and the route should be able to fall back or abstain when the evidence is insufficient.

Common Failure Modes

The Gate Validates Syntax, Not Meaning

JSON Schema can prove shape, not correctness. Add semantic checks for identifiers, policy limits, evidence provenance, and expected state transitions.

The Router Learns Provider or Dataset Artifacts

A learned router can exploit formatting, latency, or benchmark artifacts that disappear in production. Test on held-out time windows, new tools, and changed model pools.

Escalation Hides Wasted Work

A high final acceptance rate can conceal an expensive failed first pass. Report rejected-attempt cost and fallback latency separately.

The Route Changes the Agent State

Two models may summarize evidence differently or produce different tool arguments, causing later steps to diverge. Evaluate full trajectories rather than isolated calls.

A Fallback Repeats a Side Effect

Retries must be idempotent. Use operation IDs, deduplication, and a commit record before allowing a fallback to invoke a state-changing tool.

Cost Optimization Overrides Safety

Set quality and safety as constraints. Optimize cost only among policies that pass them. For unsupported or high-risk work, abstention is a valid successful policy outcome.

Production Checklist

  1. Define step and trajectory acceptance before comparing routes.
  2. Establish a simple static baseline; do not assume a learned router is better.
  3. Version the candidate set, policy, prompts, validators, and dataset.
  4. Include router, verifier, retry, fallback, tool, review, and correction cost.
  5. Calibrate thresholds by meaningful workload slice.
  6. Keep authorization and irreversible commits outside model control.
  7. Replay, shadow, canary, and rollback in that order.
  8. Trace route decisions to final outcomes without exposing sensitive data.
  9. Re-evaluate after any provider, model, prompt, tool, or workload change.
  10. Prefer abstention over forced execution when no route satisfies the contract.

Frequently Asked Questions

Should an Agent use one model for planning and another for tools?

Only if evaluation shows that the split improves accepted trajectories under the same safety and latency constraints. "Large model plans, small model executes" is a useful hypothesis, not a universal architecture. Some workflows need a stronger model for tool selection; others can use deterministic code for planning and a model only for synthesis.

Can model confidence decide when to escalate?

Confidence can be one feature, but it must be calibrated against actual acceptance outcomes. Self-reported confidence is especially weak evidence. Combine it with deterministic validation, task-specific evaluators, risk rules, and historical route performance.

Is a cascade always more accurate than a router?

No. A cascade has more evidence after the first output, but a weak verifier can accept bad work or escalate good work. It also adds latency and cost. Compare both against a static baseline on the same workload.

How often should routing policies be re-evaluated?

Re-evaluate whenever the candidate model, provider, prompt, tool schema, validator, traffic mix, or risk policy changes. Also monitor route distribution and acceptance continuously because silent provider changes and workload drift can alter calibration.

What should happen when every model route fails?

The policy should abstain, request clarification, choose a deterministic fallback, or require human approval according to the step contract. Repeatedly sending the same request to larger models is not a complete failure policy.

References

Summary

Agent model routing is an evaluation and control problem, not a model leaderboard. Define the step contract, route only among eligible candidates, validate outputs before side effects, and measure the complete trajectory. A cheaper first attempt creates value only when it lowers cost per accepted trajectory without violating quality, latency, or safety gates.