Claude 4 Opus vs Sonnet: Benchmarks and Migration

Claude Opus 4 and Claude Sonnet 4 were Anthropic's May 2025 hybrid-reasoning models. They mattered because they combined strong coding performance with reasoning between tool calls, but the original API snapshots are now retired. The durable lesson is not that either model was universally "best." It is that model, agent scaffold, compute policy, permissions, and acceptance tests form one production system.

For a new deployment, do not copy the old model IDs from a launch tutorial. Use Anthropic's current model catalog, then replay your own evaluation set before changing traffic. This guide explains the historical Claude 4 results, what they did and did not prove, and the production controls that survive model generations.

Key takeaways

• Anthropic launched Opus 4 and Sonnet 4 on May 22, 2025 with 200K-token context windows, hybrid reasoning, text and image input, tool use, and launch prices of $15/$75 and $3/$15 per million input/output tokens.
• The 72.5% and 72.7% SWE-bench Verified figures were Anthropic results under a documented harness. They were not model-only constants that transfer unchanged to another agent.
• The 79.4% and 80.2% "high compute" figures used multiple parallel attempts, visible-test rejection, and an internal scorer. They should not be compared with single-attempt results as if the inference budgets were equal.
• The often-repeated seven-hour claim came from one Rakuten open-source refactor reported by Anthropic. It demonstrated possibility, not general autonomous reliability.
• Anthropic retired both original Claude 4 snapshots from its API on June 15, 2026. Migration requires behavioral replay because model IDs, thinking controls, content blocks, cost, and refusal behavior can all change.
• Opus 4's ASL-3 deployment did not secure downstream applications. Tool allowlists, sandboxing, approvals, idempotency, audit logs, and recovery remained application responsibilities.

What Claude 4 introduced

Claude 4 was a product family, not a disclosed new neural architecture. Anthropic described both models as hybrid reasoning models with a normal response mode and an opt-in extended-thinking mode. The release also enabled tool use during extended thinking, parallel tool calls, improved use of developer-provided memory files, and new API features such as code execution, Files API, prompt caching, and a remote MCP connector.

The launch comparison is useful when read as a dated product record:

Prices and capabilities in this table are historical. Current models have different identifiers, reasoning controls, limits, and prices.

Current lifecycle status

Anthropic deprecated the two original snapshots on April 14, 2026 and retired them on June 15, 2026:

• claude-opus-4-20250514 had claude-opus-4-8 listed as its replacement.
• claude-sonnet-4-20250514 had claude-sonnet-4-6 listed as its replacement.

Those were the documented replacements for that retirement event, not permanent "latest model" aliases. Anthropic-operated and partner-operated platforms can follow different retirement schedules. Query the current model catalog for new work and pin the selected snapshot or stable identifier in configuration rather than scattering it through code.

How hybrid reasoning actually worked

Extended thinking was request configuration, not an automatic guarantee that the model would always choose the right amount of reasoning. On the original Claude 4 API, a caller enabled it with thinking.type: "enabled" and assigned a budget_tokens value. Interleaved thinking allowed the model to reason after receiving a tool result and before selecting its next action.

Three distinctions matter:

1. Thinking is not execution. In Tool Use, the model proposes calls while the application or managed API validates and executes them.
2. A thinking budget is not a quality guarantee. More tokens can improve a difficult answer, but they also add latency and cost and can still produce an incorrect trajectory.
3. A thinking summary is not an audit log. Anthropic said long Claude 4 thought processes could be summarized by a smaller model. Store tool requests, tool results, source references, state transitions, and acceptance outcomes as evidence.

The original manual-budget interface also did not become a permanent API contract. Newer Claude generations introduced adaptive thinking and different parameter rules. A migration therefore needs request-schema tests, not just a model-name substitution.

This loop, governed by an Agent Runtime, is the real unit of evaluation. Changing the model while holding the prompt, tools, permissions, and verifier constant is the minimum needed to attribute a result to the model.

Reading the Claude 4 benchmarks correctly

At launch, Anthropic reported the following coding results:

The first row did not show that Sonnet 4 was categorically better than Opus 4. A 0.2 percentage-point difference is too small to support that conclusion, especially for stochastic agents. It also did not mean either model would solve roughly three quarters of an arbitrary company's tickets. SWE-bench Verified covers 500 human-filtered issues from Python repositories, with an automated test oracle and a particular tool environment.

The high-compute row measured a different system. Sampling several candidates, rejecting patches that fail visible tests, and selecting one with an internal model spends more inference and introduces another decision component. That can be a valid production pattern, but its cost and failure modes belong in the result.

The official SWE-bench leaderboard also contains multiple results for the same model under different agents. That is direct evidence that a "model score" is partly a scaffold score. Compare entries only after aligning:

• dataset and exact task subset;
• agent version, prompts, tools, and repository access;
• sampling parameters and number of independent attempts;
• step, token, time, and monetary budgets;
• candidate rejection or selection logic;
• test visibility and contamination controls;
• provider snapshot and evaluation date.

Independent 2026 research on 60,000 SWE-bench Verified trajectories found that single-run pass@1 estimates varied by 2.2 to 6.0 percentage points across runs, with standard deviations above 1.5 points even at temperature zero. That study did not evaluate Claude 4 specifically, but it demonstrates why small leaderboard gaps should not drive a procurement decision.

What the seven-hour coding result proved

Anthropic's launch post said Rakuten validated Opus 4 on a demanding open-source refactor that ran independently for seven hours. This is useful partner evidence that the model and its harness could sustain one long workflow. It does not establish:

• a seven-hour context or uptime guarantee;
• a success rate across repositories or programming languages;
• correctness of every intermediate action;
• safe access to unrestricted shell, network, secrets, or Git operations;
• lower total engineering cost than a shorter human-supervised workflow.

Long-horizon reliability is multiplicative. If a workflow has 40 consequential steps and each step is correct 98% of the time, the probability that every step is correct is approximately 0.98^40, or 44.6%, before accounting for correlated failures. Recovery checkpoints, local verification, bounded retries, and reversible side effects matter more than elapsed time.

Memory files have the same boundary. They can preserve plans and discoveries across context pressure, but they are application-visible state that can become stale, poisoned, or contradictory. Version them, attach provenance, and validate them before reuse. For the surrounding control loop, see the Agent Loop engineering guide.

Choosing between model tiers

At launch, Sonnet 4 was the rational default candidate for many bounded tasks because its token price was one fifth of Opus 4's and its published coding result was close under Anthropic's setup. Opus 4 was a candidate for difficult tasks where measured quality, consistency, or recovery justified the premium. Neither sentence is a substitute for an evaluation.

Use a workload matrix before routing:

A practical router starts with deterministic metadata such as task class, repository size, risk level, and prior evaluation results. Enforce a versioned Context Budget, and do not let an untrusted prompt or the model's self-assessment unilaterally escalate permissions or spend.

The meaningful cost metric is:

accepted-task cost =
  (model tokens + cache operations + tool execution + retries + review time)
  / accepted outputs

A cheaper model that requires repeated repair can cost more per accepted task. An expensive model can also waste budget when the task has a simple deterministic solution. Record both token spend and end-to-end cost.

Build a task-specific evaluation

Start with an acceptance contract, not a prompt collection. Each task should define the input fixture, permitted context, tool policy, required checks, forbidden changes, and evidence needed for acceptance.

1. Sample production work

Stratify cases by difficulty and consequence:

• localized bug fixes with deterministic tests;
• multi-file refactors with compatibility constraints;
• ambiguous incidents that require evidence gathering;
• tool workflows with read and write operations;
• adversarial inputs containing prompt injection or misleading repository text.

Keep a private holdout set. Public benchmark overlap can inflate confidence, and a benchmark made only of clean, solvable issues will underrepresent production ambiguity.

2. Repeat trials

Run each candidate several times per task with the same harness and budgets. Report the mean and dispersion, not only the best run. Preserve failed trajectories because they reveal repeated misunderstandings, unsafe actions, and expensive loops.

3. Separate outcome dimensions

Track at least:

An LLM-as-judge can help triage subjective outputs, but it should be calibrated against human labels and never replace executable tests for deterministic requirements.

4. Define release gates before seeing results

Set thresholds by task class. A documentation draft may tolerate a retry; a migration or payment change should require deterministic validation and human approval. Locking thresholds before comparison reduces the temptation to select whichever metric favors a preferred model.

A runnable Go evaluation gate

The following program consumes trial records from standard input, groups them by model, and fails a candidate unless it meets acceptance, latency, safety, and accepted-cost thresholds. In a real harness, accepted should be produced by tests and policy checks rather than by the model itself.

package main

import (
	"encoding/json"
	"fmt"
	"os"
	"sort"
)

type Trial struct {
	Model            string  `json:"model"`
	Task             string  `json:"task"`
	Accepted         bool    `json:"accepted"`
	LatencyMS        int64   `json:"latency_ms"`
	CostUSD          float64 `json:"cost_usd"`
	PolicyViolations int     `json:"policy_violations"`
}

type Report struct {
	Model              string  `json:"model"`
	Trials             int     `json:"trials"`
	AcceptanceRate     float64 `json:"acceptance_rate"`
	P95LatencyMS       int64   `json:"p95_latency_ms"`
	CostPerAcceptedUSD float64 `json:"cost_per_accepted_usd"`
	PolicyViolations   int     `json:"policy_violations"`
	Passed             bool    `json:"passed"`
}

func p95(values []int64) int64 {
	sort.Slice(values, func(i, j int) bool { return values[i] < values[j] })
	index := (95*len(values)+99)/100 - 1
	return values[index]
}

func summarize(model string, trials []Trial) Report {
	accepted := 0
	totalCost := 0.0
	violations := 0
	latencies := make([]int64, 0, len(trials))

	for _, trial := range trials {
		if trial.Accepted {
			accepted++
		}
		totalCost += trial.CostUSD
		violations += trial.PolicyViolations
		latencies = append(latencies, trial.LatencyMS)
	}

	rate := float64(accepted) / float64(len(trials))
	costPerAccepted := -1.0
	if accepted > 0 {
		costPerAccepted = totalCost / float64(accepted)
	}

	return Report{
		Model:              model,
		Trials:             len(trials),
		AcceptanceRate:     rate,
		P95LatencyMS:       p95(latencies),
		CostPerAcceptedUSD: costPerAccepted,
		PolicyViolations:   violations,
		Passed:             rate >= 0.80 && p95(latencies) <= 30_000 && costPerAccepted >= 0 && costPerAccepted <= 1.50 && violations == 0,
	}
}

func main() {
	var trials []Trial
	if err := json.NewDecoder(os.Stdin).Decode(&trials); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(2)
	}
	if len(trials) == 0 {
		fmt.Fprintln(os.Stderr, "no trials")
		os.Exit(2)
	}

	grouped := make(map[string][]Trial)
	for _, trial := range trials {
		grouped[trial.Model] = append(grouped[trial.Model], trial)
	}

	models := make([]string, 0, len(grouped))
	for model := range grouped {
		models = append(models, model)
	}
	sort.Strings(models)

	reports := make([]Report, 0, len(models))
	allPassed := true
	for _, model := range models {
		report := summarize(model, grouped[model])
		reports = append(reports, report)
		allPassed = allPassed && report.Passed
	}

	encoder := json.NewEncoder(os.Stdout)
	encoder.SetIndent("", "  ")
	if err := encoder.Encode(reports); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(2)
	}
	if !allPassed {
		os.Exit(1)
	}
}

Keep the gate definition versioned with the dataset and harness. A report without the model snapshot, prompt hash, tool versions, trial count, and evaluator version is not reproducible evidence.

Production controls beyond model quality

A strong model can increase blast radius because it can operate tools more effectively. Treat every tool result as untrusted input and every write as a controlled side effect.

Permission and execution controls

• Expose the minimum tool set for the current task; use allowlists for MCP and native tools.
• Separate read, prepare, approve, and commit capabilities.
• Run generated commands in a sandbox with scoped filesystem and network access.
• Keep secrets out of prompts and tool output; issue short-lived, task-scoped credentials.
• Require human approval for destructive, external, financial, identity, or release actions.
• Cap steps, tokens, elapsed time, retries, and spend independently.

The MCP connector is an API integration surface, not a trust shortcut. It supports remote HTTP servers and tool calls, while local STDIO servers, prompts, and resources require a client-side implementation. Pin or review tool definitions, because changed descriptions or schemas can alter model behavior.

State and recovery controls

• Assign an idempotency key to each external mutation.
• Persist checkpoints before and after side effects.
• Record tool arguments, normalized results, approvals, and commit receipts.
• Resume from verified state instead of replaying the whole conversation.
• Stop when evidence is contradictory or the acceptance oracle is unavailable.

These controls are explained in more depth in the Agent Loop guide and Prompt Injection glossary.

What ASL-3 did and did not mean

Anthropic launched Opus 4 under its ASL-3 Deployment and Security Standards as a precaution. The company said it had not determined that Opus 4 definitively crossed the relevant CBRN capability threshold, but could not confidently rule that risk out. The measures focused on narrowly scoped CBRN misuse defenses and stronger protection against model-weight theft. Sonnet 4 remained under ASL-2.

ASL-3 did not mean:

• every Opus 4 answer was correct or harmless;
• an application inherited secure authentication, authorization, or tenant isolation;
• unrestricted computer or shell use was safe;
• prompt injection was solved;
• the model's reasoning summary was a faithful compliance record.

The Claude 4 system card documented reward hacking, prompt-injection tests, high-agency behavior, and concerning actions in artificial alignment scenarios. These evaluations are reasons to constrain agent authority, not evidence that ordinary deployments will reproduce every scenario. Security claims should stay attached to their test setup and threat model.

Migration checklist for retired Claude 4 workloads

Changing only the model string is insufficient. Use this sequence:

1. Inventory usage. Export provider usage by API key and model, then search source, configuration, queues, saved prompts, and fallback rules for retired IDs.
2. Freeze a baseline. Save representative requests, tool transcripts, accepted outputs, costs, latency, refusals, and incident examples from existing logs.
3. Select an active target. Check the current catalog for the exact platform and region. Do not assume Anthropic API, Bedrock, and Vertex AI share lifecycle dates or IDs.
4. Adapt request semantics. Verify thinking mode, supported parameters, beta headers, maximum output, tool definitions, and content-block parsing.
5. Replay the acceptance suite. Run repeated trials under equal budgets. Compare correctness, consistency, accepted-task cost, tool failures, and policy behavior.
6. Canary by task class. Shift low-risk traffic first, retain an explicit fallback, and monitor schema errors, refusals, latency, spend, and human overrides.
7. Remove dead compatibility paths. Delete retired IDs only after queues, scheduled jobs, disaster recovery, and rollback documentation point to supported models.

For model-independent design, keep a provider adapter around normalized messages, tool calls, usage, finish reasons, and errors. Keep model-specific reasoning and beta flags at the adapter boundary.

Common mistakes

Treating a launch chart as a current leaderboard

Benchmark entries age as models, datasets, and agents change. Preserve the release date and methodology whenever quoting a score.

Comparing pass@1 with best-of-many

Parallel attempts plus candidate selection can improve success while multiplying cost. Label the inference policy and report total spend.

Equating context capacity with reliable memory

A 200K window is an input limit, not proof that every detail will be retrieved or obeyed. Test long-context recall, conflicts, and distractors on your own documents.

Giving a coding agent full access because the model is capable

Capability is not authorization. Keep permission checks, sandboxing, approvals, and irreversible commits outside the model.

Parsing only the first response block

Thinking and tool-enabled responses can contain several block types. Dispatch by block type, preserve required tool-loop state, and fail closed on unknown structures.

FAQ

Are Claude Opus 4 and Claude Sonnet 4 still available through the Anthropic API?

No. Anthropic retired claude-opus-4-20250514 and claude-sonnet-4-20250514 on June 15, 2026. Partner-operated platforms can use different schedules, so verify the provider catalog rather than inferring availability from an old tutorial.

What did the Claude 4 SWE-bench scores measure?

Anthropic reported 72.5% for Opus 4 and 72.7% for Sonnet 4 as pass@1 averages over ten trials on all 500 SWE-bench Verified tasks, using bash and editor tools without test-time compute. The separate 79.4% and 80.2% results added parallel attempts, visible-test filtering, and internal candidate scoring.

Did Claude Opus 4 reliably work autonomously for seven hours?

No general guarantee was established. The launch post cited one Rakuten open-source refactor that ran independently for seven hours. A production team still needs checkpoints, deterministic tests, budgets, supervision rules, and recovery drills.

Was Claude Opus 4 safer because it used ASL-3 protections?

ASL-3 covered Anthropic's deployment controls for narrowly scoped CBRN misuse and security controls for model weights. It was not an application-security certification and did not remove risks from tools, credentials, data, or prompt injection.

How should a team choose between Sonnet- and Opus-tier models?

Start with a representative acceptance suite. Use the lower-cost tier when it meets quality and reliability thresholds; escalate when the higher tier produces enough measurable lift to justify its cost and latency. Compare repeated runs and cost per accepted task rather than one headline score.

Sources

• Anthropic: Introducing Claude 4
• Anthropic: Claude 4 System Card
• Anthropic: Model deprecations
• Anthropic: Models overview
• Anthropic: Extended thinking
• Anthropic: MCP connector
• Anthropic: Activating AI Safety Level 3 protections
• SWE-bench: Official leaderboards
• Bjarnason, Silva, and Monperrus: On Randomness in Agentic Evals

Related reading

• Reasoning Models: OpenAI o1 and DeepSeek-R1
• Agent Loop: State, Tools, Recovery, and Evaluation
• MCP Protocol: Architecture and Security
• Context Engineering: Four-Layer Architecture