TL;DR
Freeze tasks, environments, acceptance, and budget rules, then cross models with harness configurations. Compare outcomes on matched tasks, ablate components, and measure interaction. Equal-budget tests estimate constrained efficiency; cost curves expose tradeoffs. More successful attempts do not automatically establish a better architecture.
Before celebrating a gain, inspect what changed
Changing the model, adding browser tools, and increasing retries from one to five creates an attribution problem. Higher success may be useful, but it does not identify the contribution of the harness.
An agent harness includes tool interfaces, context policies, scheduling, retries, and acceptance. “Same model” also needs a version, sampling configuration, and system instructions. Two commercial products advertising the same model may still use different hidden settings.
Anthropic's 2026 application-development experiment compares single-agent and full-harness outputs. One example contrasts approximately 20 minutes and $9 with six hours and $200. The artifacts are informative, but the budgets differ substantially. They do not establish equal-cost improvement, and the figures are author-reported runs rather than standard prices or independently reproduced benchmarks.
Write the question before running the experiment. “Does structured evaluation feedback improve acceptance under the same per-task spending cap?” is testable. “Is our new architecture stronger?” leaves too many variables unspecified.
Cross models and harnesses in four cells
Running two models under two harnesses separates model differences, harness differences, and their interaction.
| Configuration | Existing harness H0 | New harness H1 |
|---|---|---|
| Model M0 | Y00 | Y01 |
| Model M1 | Y10 | Y11 |
Y represents average acceptance over the same tasks. Freeze initial snapshots, grading, and budget rules. For a pass-rate outcome:
Harness gain on M0 = Y01 - Y00
Harness gain on M1 = Y11 - Y10
Interaction = (Y11 - Y10) - (Y01 - Y00)
A large interaction means the harness effect depends on the model. One model may need frequent context resets while another can continue after compaction. Tool descriptions can have different effects across models. A favorable cell does not establish a universal policy.
Run each task in all four cells to obtain paired observations. Randomize execution order so that one group does not systematically encounter outages or service updates. Record timestamps and observable versions for external services that cannot be frozen.
Pairing controls task composition, not every source of variation. Repetitions and order randomization still matter. If the model endpoint changes during the experiment, split or rerun the affected block rather than silently treating the results as one stationary system.
Budget fairness needs constrained results and curves
An equal-budget comparison and a quality-cost curve answer different questions. Both are useful.
| Comparison | Fixed condition | Supported conclusion |
|---|---|---|
| Equal spending cap | Total per-task cost, including failures and evaluation | Which system works better under that cap |
| Equal time cap | Wall-clock time and concurrency rules | Which meets the latency constraint |
| Multiple budget points | Increasing spending or time limits | Whether additional quality justifies the expense |
| Native defaults | Each product's default policy | Default experience, with limited attribution |
Cost includes generation, evaluation, tools, and failed retries. Time includes waiting and queues. A task that exhausts its cap without acceptance remains failed or incomplete; do not remove it from the denominator.
Report cost distributions for all tasks and a separate successful-task view. Otherwise expensive failures disappear through survivorship bias. Total cost divided by accepted tasks is a useful cost-per-success measure, but it is undefined when there are no successes and depends on a shared acceptance definition.
An equal dollar cap does not imply equal token counts or latency. Record those secondary outcomes. If one harness uses more parallel workers, a time comparison must explicitly declare concurrency and its cost consequences.
Ablate components with interpretable controls
Ablation disables one component and measures the change. It estimates value in the existing combination, not necessarily value in isolation.
Suppose feedback depends on browser execution. Disabling the browser also deprives the evaluator of observations. A weak result does not establish that evaluation is useless. Cross browser availability and feedback availability to test the dependency.
When removing feedback, an equal-budget resampling control distinguishes extra inference from useful defect information. That experiment estimates the value of an alternative spending policy. Simply removing the evaluator estimates the effect of deleting the component from the existing system. Both questions are legitimate, but they are not interchangeable.
Failure slices often guide implementation better than the aggregate score. Fewer tool-selection errors with more timeouts suggests improved decisions at a latency cost. An agent trajectory provides the observations needed to diagnose such changes.
Do not replace missing components with broken stubs. An ablated configuration should still execute its intended policy correctly. Otherwise the experiment measures an implementation defect instead of the component's contribution.
Runnable Go: paired gains and interaction
This Go 1.23 program uses four synthetic tasks, each containing outcomes for all four cells. It demonstrates the calculation, not actual model performance.
package main
import "fmt"
type Task struct {
ID string
Pass [4]int
}
func analyze(tasks []Task) ([4]float64, error) {
var means [4]float64
if len(tasks) == 0 {
return means, fmt.Errorf("empty task set")
}
seen := make(map[string]bool)
for _, task := range tasks {
if task.ID == "" || seen[task.ID] {
return means, fmt.Errorf("missing or duplicate task identity")
}
seen[task.ID] = true
for cell, value := range task.Pass {
if value != 0 && value != 1 {
return means, fmt.Errorf("invalid outcome for %s", task.ID)
}
means[cell] += float64(value)
}
}
for cell := range means {
means[cell] /= float64(len(tasks))
}
return means, nil
}
func main() {
tasks := []Task{
{ID: "t1", Pass: [4]int{0, 1, 1, 1}},
{ID: "t2", Pass: [4]int{0, 0, 0, 1}},
{ID: "t3", Pass: [4]int{1, 1, 1, 1}},
{ID: "t4", Pass: [4]int{0, 0, 0, 1}},
}
y, err := analyze(tasks)
if err != nil {
panic(err)
}
gainM0 := y[1] - y[0]
gainM1 := y[3] - y[2]
fmt.Printf("cells=%.2f %.2f %.2f %.2f\n", y[0], y[1], y[2], y[3])
fmt.Printf("harness_gain M0=%.2f M1=%.2f interaction=%.2f\n",
gainM0, gainM1, gainM1-gainM0)
}
Run go run main.go:
cells=0.25 0.50 0.50 1.00
harness_gain M0=0.25 M1=0.50 interaction=0.25
The array order is M0H0, M0H1, M1H0, M1H1. The new harness gains 25 percentage points on M0 and 50 on M1, producing a 25-point interaction. Four tasks are insufficient for a significance claim.
A real experiment repeats every cell, calculates per-task averages, and aggregates paired differences. Estimate intervals with task-level bootstrap and disclose sample size and sampling procedure. The repeated reliability article covers repetition metrics. Treating every trial as an independent task understates uncertainty from task clustering.
The example checks task identity and outcome validity, but does not enforce budget equality. A production runner must validate configuration manifests and account for spending before analysis. An elegant calculation cannot rescue mismatched experimental inputs.
Preserve the conditions in the report
A reviewable report needs task and environment versions, four configurations, budget rules, graders, repetition counts, paired differences, and failure slices. A mean score alone cannot tell another team whether to adopt the change.
Declare primary outcomes and important slices before inspecting results. Use development tasks to tune policies and untouched tasks for final comparisons. Repeatedly selecting configurations on the same test set turns that set into development data.
A supported conclusion might say: “On this code-task set, model version, and spending cap, structured feedback improved acceptance while increasing median latency.” Removing those conditions and writing “feedback harnesses are universally better” exceeds the experiment.
The evaluator feedback-loop article explains the mechanism being tested. Mechanism descriptions and measured effects should remain distinct: a plausible explanation does not replace a controlled result.
FAQ
What if commercial configurations are opaque?
Define the study as a default-product comparison and document visible settings. Compare outcomes and costs without claiming to isolate hidden prompts or model weights.
Can ablation effects be added together?
Not automatically. Components can complement or replace each other. Measure important combinations unless the experiment supports an approximately additive model.
Should timeouts be excluded?
System timeouts belong in the primary result. Infrastructure incidents can support a separate sensitivity analysis, but exclusion rules should be predefined and original results retained.
Does winning at one budget justify migration?
Check task coverage, severe failures, latency, and maintenance cost. Validate several budget points and run a small test in the intended workflow before generalizing.
References
- Anthropic: Harness design for long-running application development, March 24, 2026.
- Anthropic: Demystifying evals for AI agents, January 9, 2026.