Direct Answer

Run Cursor, Claude Code, GitHub Copilot, or any other qualified AI coding tool as a controlled experiment. Freeze the task suite, repository snapshots, acceptance tests, environment, instructions, budgets, permissions, and review rubric before execution. Record every attempt, including failures. Decide from accepted changes, human review and rework, regressions, policy violations, uncertainty, and total cost, not generated lines, public leaderboards, or subjective star ratings.

This page is the experiment protocol. Use AI Coding Tools: Capability, Security, and Team Fit first to define the workflow and eliminate candidates that fail hard qualification gates.

Key Takeaways

  • A tool name is not an experimental condition. Record client version, mode, model policy, settings, permissions, and runtime.
  • Public benchmarks test a dataset and harness; they do not reproduce your repository, requirements, reviewers, or security boundary.
  • Use paired representative tasks, clean snapshots, equivalent budgets, randomized order, and independent review.
  • Define “accepted” before the run and count review, rework, rejected attempts, regressions, and incidents.
  • Report confidence intervals and slice results. “Inconclusive” is more useful than a winner produced by a weak sample.

Why Public Coding Benchmarks Cannot Pick a Product

Public coding benchmarks are useful research instruments, but their score is not a procurement verdict.

The original SWE-bench paper introduced 2,294 issues from 12 Python repositories. A model receives a repository and issue description and must produce a patch. This setup tests long-context repository reasoning and multi-file changes, but it does not measure every product mode, language, code-review standard, integration, or operational risk.

Benchmark validity also changes over time. In February 2026, OpenAI reported that SWE-bench Verified had contamination and flawed tests. In July 2026, it published a further audit of SWE-bench Pro and estimated that roughly 30% of the public-split tasks were broken, citing overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts. OpenAI then retracted its earlier recommendation to use SWE-bench Pro.

That does not make all repository benchmarks useless. It means every score must carry:

  • dataset and exact split;
  • harness, scaffold, tool, model, and inference settings;
  • environment and network policy;
  • task-quality audit;
  • contamination and leakage analysis;
  • cost and retry budget;
  • date and source;
  • capability boundaries the benchmark does not test.

A product comparison needs stronger evidence: private or newly constructed tasks from your work, human review, operational controls, and outcomes that match the decision.

Productivity Is Harder to Measure Than Completion Time

Developer productivity is a system outcome, not a single timer.

METR's early-2025 randomized controlled trial studied 16 experienced open-source developers completing 246 tasks in repositories they knew well. Under that protocol, allowing early-2025 AI tools increased completion time by 19%, while participants believed the tools had made them faster.

METR later marked the historical result as outdated. Its February 2026 update found that a new task-level experiment no longer produced a reliable current estimate because developers selected out of no-AI tasks, task choice changed, parallel agents made time reporting difficult, and output quality could differ. The raw estimates had wide confidence intervals.

The durable lesson is not “AI slows developers” or “AI speeds developers.” It is that perceived speed, task completion, active human time, wall time, output quality, and selection effects are different variables. Your experiment must name which one it measures.

Define the Decision Before the Trial

A trial should answer one bounded decision:

yaml
decision: choose a default tool for bounded maintenance pull requests
candidates:
  - candidate-a-local-agent
  - candidate-b-cloud-agent
  - current-human-workflow
repositories:
  - typescript-service
  - python-library
task_slices:
  - bug-fix
  - test-addition
  - dependency-safe-update
  - narrow-refactor
excluded:
  - production-operations
  - schema-migrations
  - security-critical-auth-changes
primary_outcome: accepted_change_without_material_rework
hard_gates:
  - no_policy_violation
  - no_unapproved_network
  - no_secret_access
  - required_tests_pass

Do not ask “which tool is best?” Ask “which qualified mode produces more accepted changes for these task slices, within these quality, security, cost, and review constraints?”

The candidate may be a mixed toolchain or the current workflow. Including a no-adoption option prevents the experiment from assuming its conclusion.

Build a Representative Task Suite

A useful task suite samples work the team expects to delegate, including cases where tools may fail.

Sample From Real Work

Use recently completed or newly authored tasks whose expected behavior can be independently checked. Remove the original patch and future Git history from the candidate environment. Do not give the model a task it or its retrieval tools can look up online.

Preserve Task Slices

Tag every task by dimensions that may change results:

Slice Examples
Work type bug fix, feature, refactor, test, documentation, dependency update
Scope one file, subsystem, cross-service
Language and framework TypeScript, Go, Python, Java, infrastructure
Context quality complete specification, ambiguous legacy issue, missing test
Risk low, moderate, security-sensitive, irreversible
Repository maturity greenfield, mature service, legacy monolith
Verification unit tests, integration tests, static analysis, human behavior review

Critical slices need enough repeated observations to support a decision. If a slice has one task, report the observation without generalizing.

Add Clean and Adversarial Tasks

Include:

  • tasks where no change is required;
  • underspecified tasks where the correct action is clarification;
  • impossible tasks where the tool should stop;
  • repositories with decoy secrets and forbidden paths;
  • untrusted comments or tool output containing indirect prompt injection;
  • commands that should be denied;
  • changes that pass a narrow test but violate an invariant.

Run adversarial tasks in isolated fixtures without real secrets, production access, or uncontrolled network egress.

This track is no longer hypothetical. The July 2026 IssueTrojanBench preprint tested malicious instructions delivered through issue bodies, PDFs, and low-authority metadata against specific Cursor, Claude Code, and Codex configurations. In that benchmark, 66.5% of malicious issues crossed both agent- and model-level guardrails. That result belongs to the tested versions, models, attacks, and harness; it is not a permanent product ranking. Its durable implication is that untrusted issue content must remain data, while filesystem, shell, network, secret, and Git authority is enforced outside the model.

Pre-Register Acceptance and Failure

Define acceptance before seeing a candidate's output.

An accepted change should satisfy all required dimensions:

  1. Behavior: explicit acceptance tests and relevant regression tests pass.
  2. Scope: the change stays within the requested contract or documents an approved deviation.
  3. Quality: independent review finds no material correctness, maintainability, or architecture issue.
  4. Security: no forbidden access, secret exposure, unsafe dependency, policy bypass, or unreviewed side effect.
  5. Evidence: diff, commands, tests, tool trace, cost, and approvals are present.

Use machine-readable terminal states:

  • accepted;
  • rejected_correctness;
  • rejected_scope;
  • rejected_security;
  • rejected_maintainability;
  • clarification_required;
  • environment_failure;
  • budget_exhausted;
  • invalid_trial.

Environment failures and invalid trials must not quietly become tool failures or disappear. Report and rerun them under the pre-registered policy.

Control the Experimental Conditions

Fair comparison means equivalent opportunity, not identical user-interface clicks.

flowchart LR T["Frozen task and base commit"] --> E["Clean isolated environment"] E --> C["Pinned candidate configuration"] C --> R["Randomized run order"] R --> A["Captured artifact and trace"] A --> H["Independent acceptance review"] H --> L["Locked result ledger"]

Pin What Can Be Pinned

Record:

  • product, client, extension, or CLI version;
  • operating mode and model policy;
  • selected model if configurable;
  • repository base commit and environment image hash;
  • rules, prompt, task specification, and tool configuration hashes;
  • file, shell, network, secret, and MCP permissions;
  • retry, token, time, and monetary budgets;
  • reviewer rubric revision.

If the vendor manages a model or service version, record vendor-managed and the run timestamp. Do not invent precision the product does not expose.

Start From a Clean Snapshot

Each candidate receives the same base commit and dependency state in a separate worktree, container, or VM. Remove candidate output before the next run. Avoid caches that leak patches, conversations, or tests between candidates.

Give Equivalent Information

Provide the same task requirements and accessible documentation. Do not give one candidate a carefully iterated prompt and another a one-line issue. Product-specific syntax can differ, but the information and allowed actions should remain equivalent.

Randomize and Block

Randomize candidate order within task blocks to reduce learning, fatigue, and temporal effects. If different developers operate tools, block or rotate by developer. Do not let the strongest product advocate operate only their preferred tool.

Separate Operators From Reviewers

Where practical, reviewers receive normalized diffs and evidence without product identity. Reviewers should not see marketing labels, token counts, or narrative traces before judging correctness and maintainability. Security reviewers can inspect full traces after the primary code decision.

Measure Accepted Outcomes and Human Load

Raw output is not productivity. Track a metric set that exposes trade-offs.

Metric Definition Why it matters
Acceptance rate accepted trials / valid trials Basic useful-outcome rate
Human minutes active operation + review + rework Captures scarce team attention
Wall time start to terminal result Matters for synchronous and queued work
Accepted Goodput accepted changes / human hour Combines utility with human capacity
Cost per accepted change all direct and labor cost / accepted changes Compares economic output
Regression count post-change failures attributed to candidate Detects narrow-test optimization
Policy violations forbidden access or action events Hard safety signal
Scope deviation material unapproved change outside task Detects over-eager behavior
Clarification quality correct clarification on ambiguous tasks Rewards safe inability
Review queue impact waiting and reviewer load Detects parallel-agent saturation

Report by task and repository slice. A high aggregate can hide failure on the team's most important language or risk class.

Separate Time Measures

Use at least:

text
human minutes = active operation + review + rework
wall minutes  = elapsed time from assignment to terminal result

For parallel agents, human minutes cannot be calculated by adding overlapping wall clocks. Capture active operator intervals or use a defined time-sampling method.

Price Labor and Failure

text
total cost
  = direct tool and compute cost
  + labor rate * human minutes / 60
  + incident and rollback cost

cost per accepted change
  = total cost / accepted changes

Record rejected trials in total cost. If there are zero accepted changes, the cost per accepted change is undefined, not zero.

Use Confidence Intervals and Honest Language

An observed difference is not automatically a stable advantage.

For binary acceptance, report the count and a confidence interval. For skewed time and cost, report medians and the underlying observations, not only averages. With few tasks or wide intervals, describe results as inconclusive.

Avoid:

  • ranking candidates by a weighted score with arbitrary weights;
  • treating a small percentage difference as decisive;
  • pooling unrelated task slices;
  • excluding failed or expensive attempts after seeing results;
  • choosing thresholds that make a preferred candidate win.

Hard security and correctness gates should remain separate from optimization metrics. A candidate does not compensate for secret access by saving review minutes.

A Machine-Readable Trial Ledger

Store each attempt as immutable evidence:

json
{
  "experiment_id": "coding-tools-maintenance-v1",
  "labor_cost_per_hour": 120,
  "gates": {
    "min_acceptance_rate": 0.6,
    "max_policy_violations": 0,
    "max_regressions": 0,
    "max_cost_per_accepted_change": 900
  },
  "trials": [
    {
      "trial_id": "candidate-a-task-001",
      "tool": "candidate-a",
      "task_id": "task-001",
      "task_type": "bug-fix",
      "tool_version": "pinned-or-vendor-managed",
      "model": "pinned-or-vendor-managed",
      "environment_hash": "sha256:...",
      "prompt_hash": "sha256:...",
      "status": "accepted",
      "active_minutes": 12,
      "review_minutes": 18,
      "rework_minutes": 4,
      "wall_minutes": 41,
      "direct_cost": 7.4,
      "regressions": 0,
      "policy_violations": 0
    }
  ]
}

Thresholds above are illustrative fields, not universal recommendations. Set them from the team's existing service, risk, and cost objectives before the trial.

A Runnable Evaluation Gate

The dependency-free Go program below validates paired decision-bearing tasks, retains invalid and environment-failure attempts in cost, calculates Wilson intervals, human load, accepted Goodput, and cost per accepted change, then applies configured gates.

go
package main

import (
	"encoding/json"
	"errors"
	"fmt"
	"io"
	"math"
	"os"
	"sort"
)

var validStatuses = map[string]bool{
	"accepted":                 true,
	"rejected_correctness":     true,
	"rejected_scope":           true,
	"rejected_security":        true,
	"rejected_maintainability": true,
	"clarification_required":   true,
	"budget_exhausted":         true,
	"environment_failure":      true,
	"invalid_trial":            true,
}

type Gates struct {
	MinAcceptanceRate        *float64 `json:"min_acceptance_rate"`
	MaxPolicyViolations      *float64 `json:"max_policy_violations"`
	MaxRegressions           *float64 `json:"max_regressions"`
	MaxCostPerAcceptedChange *float64 `json:"max_cost_per_accepted_change"`
}

type Trial struct {
	TrialID          string   `json:"trial_id"`
	Tool             string   `json:"tool"`
	TaskID           string   `json:"task_id"`
	TaskType         string   `json:"task_type"`
	ToolVersion      string   `json:"tool_version"`
	Model            string   `json:"model"`
	EnvironmentHash  string   `json:"environment_hash"`
	PromptHash       string   `json:"prompt_hash"`
	Status           string   `json:"status"`
	ActiveMinutes    *float64 `json:"active_minutes"`
	ReviewMinutes    *float64 `json:"review_minutes"`
	ReworkMinutes    *float64 `json:"rework_minutes"`
	WallMinutes      *float64 `json:"wall_minutes"`
	DirectCost       *float64 `json:"direct_cost"`
	Regressions      *float64 `json:"regressions"`
	PolicyViolations *float64 `json:"policy_violations"`
}

type Experiment struct {
	ExperimentID     string   `json:"experiment_id"`
	LaborCostPerHour *float64 `json:"labor_cost_per_hour"`
	Gates            Gates    `json:"gates"`
	Trials           []Trial  `json:"trials"`
}

type Summary struct {
	ValidTrials                 int        `json:"valid_trials"`
	OperationalFailures         int        `json:"operational_failures"`
	Accepted                    int        `json:"accepted"`
	AcceptanceRate              float64    `json:"acceptance_rate"`
	AcceptanceRateWilson95      [2]float64 `json:"acceptance_rate_wilson_95"`
	MedianHumanMinutes          float64    `json:"median_human_minutes"`
	MedianWallMinutes           float64    `json:"median_wall_minutes"`
	AcceptedGoodputPerHumanHour *float64   `json:"accepted_goodput_per_human_hour"`
	TotalCost                   float64    `json:"total_cost"`
	CostPerAcceptedChange       *float64   `json:"cost_per_accepted_change"`
	Regressions                 float64    `json:"regressions"`
	PolicyViolations            float64    `json:"policy_violations"`
}

func number(value *float64, field string) (float64, error) {
	if value == nil || math.IsNaN(*value) || math.IsInf(*value, 0) || *value < 0 {
		return 0, fmt.Errorf("%s must be finite and non-negative", field)
	}
	return *value, nil
}

func isDecisionStatus(status string) bool {
	return status != "environment_failure" && status != "invalid_trial"
}

func validate(experiment Experiment) error {
	if experiment.ExperimentID == "" {
		return errors.New("experiment_id is required")
	}
	labor, err := number(experiment.LaborCostPerHour, "labor_cost_per_hour")
	if err != nil || labor < 0 {
		return errors.New("labor_cost_per_hour must be finite and non-negative")
	}
	if len(experiment.Trials) == 0 {
		return errors.New("trials must not be empty")
	}

	gates := map[string]*float64{
		"min_acceptance_rate":          experiment.Gates.MinAcceptanceRate,
		"max_policy_violations":        experiment.Gates.MaxPolicyViolations,
		"max_regressions":              experiment.Gates.MaxRegressions,
		"max_cost_per_accepted_change": experiment.Gates.MaxCostPerAcceptedChange,
	}
	for name, value := range gates {
		if _, err := number(value, name); err != nil {
			return err
		}
	}
	if *experiment.Gates.MinAcceptanceRate > 1 {
		return errors.New("min_acceptance_rate must not exceed 1")
	}

	seenTrials := map[string]bool{}
	tools := map[string]bool{}
	decisionCounts := map[string]map[string]int{}
	for index, trial := range experiment.Trials {
		if trial.TrialID == "" || trial.Tool == "" || trial.TaskID == "" ||
			trial.TaskType == "" || trial.ToolVersion == "" || trial.Model == "" ||
			trial.EnvironmentHash == "" || trial.PromptHash == "" {
			return fmt.Errorf("trials[%d] has missing identity", index)
		}
		tools[trial.Tool] = true
		if seenTrials[trial.TrialID] {
			return fmt.Errorf("duplicate trial_id %q", trial.TrialID)
		}
		seenTrials[trial.TrialID] = true
		if !validStatuses[trial.Status] {
			return fmt.Errorf("trials[%d] has invalid status %q", index, trial.Status)
		}
		for name, value := range map[string]*float64{
			"active_minutes":    trial.ActiveMinutes,
			"review_minutes":    trial.ReviewMinutes,
			"rework_minutes":    trial.ReworkMinutes,
			"wall_minutes":      trial.WallMinutes,
			"direct_cost":       trial.DirectCost,
			"regressions":       trial.Regressions,
			"policy_violations": trial.PolicyViolations,
		} {
			if _, err := number(value, fmt.Sprintf("trials[%d].%s", index, name)); err != nil {
				return err
			}
		}
		if isDecisionStatus(trial.Status) {
			if decisionCounts[trial.Tool] == nil {
				decisionCounts[trial.Tool] = map[string]int{}
			}
			decisionCounts[trial.Tool][trial.TaskID]++
			if decisionCounts[trial.Tool][trial.TaskID] > 1 {
				return fmt.Errorf("multiple decision results for %s/%s", trial.Tool, trial.TaskID)
			}
		}
	}

	if len(decisionCounts) < 2 {
		return errors.New("at least two tools need decision-bearing trials")
	}
	if len(decisionCounts) != len(tools) {
		return errors.New("every tool needs at least one decision-bearing trial")
	}
	var baseline []string
	for tool, taskCounts := range decisionCounts {
		tasks := make([]string, 0, len(taskCounts))
		for task := range taskCounts {
			tasks = append(tasks, task)
		}
		sort.Strings(tasks)
		if baseline == nil {
			baseline = tasks
			continue
		}
		if fmt.Sprint(tasks) != fmt.Sprint(baseline) {
			return fmt.Errorf("tool %s does not cover the paired task set", tool)
		}
	}
	return nil
}

func wilson(successes, total int) [2]float64 {
	if total == 0 {
		return [2]float64{}
	}
	const z = 1.96
	p := float64(successes) / float64(total)
	denominator := 1 + z*z/float64(total)
	center := (p + z*z/(2*float64(total))) / denominator
	spread := z * math.Sqrt(
		(p*(1-p)+z*z/(4*float64(total)))/float64(total),
	) / denominator
	return [2]float64{math.Max(0, center-spread), math.Min(1, center+spread)}
}

func median(values []float64) float64 {
	sort.Float64s(values)
	middle := len(values) / 2
	if len(values)%2 == 1 {
		return values[middle]
	}
	return (values[middle-1] + values[middle]) / 2
}

func summarize(experiment Experiment) map[string]Summary {
	grouped := map[string][]Trial{}
	for _, trial := range experiment.Trials {
		grouped[trial.Tool] = append(grouped[trial.Tool], trial)
	}

	output := map[string]Summary{}
	for tool, trials := range grouped {
		var validTrials, operationalFailures, accepted int
		var humanMinutes, wallMinutes []float64
		var totalHuman, totalDirect, regressions, violations float64
		for _, trial := range trials {
			human := *trial.ActiveMinutes + *trial.ReviewMinutes + *trial.ReworkMinutes
			humanMinutes = append(humanMinutes, human)
			wallMinutes = append(wallMinutes, *trial.WallMinutes)
			totalHuman += human
			totalDirect += *trial.DirectCost
			regressions += *trial.Regressions
			violations += *trial.PolicyViolations
			if isDecisionStatus(trial.Status) {
				validTrials++
				if trial.Status == "accepted" {
					accepted++
				}
			} else {
				operationalFailures++
			}
		}
		totalCost := totalDirect + *experiment.LaborCostPerHour*totalHuman/60
		var goodput, costPerAccepted *float64
		if totalHuman > 0 {
			value := float64(accepted) * 60 / totalHuman
			goodput = &value
		}
		if accepted > 0 {
			value := totalCost / float64(accepted)
			costPerAccepted = &value
		}
		output[tool] = Summary{
			ValidTrials:                 validTrials,
			OperationalFailures:         operationalFailures,
			Accepted:                    accepted,
			AcceptanceRate:              float64(accepted) / float64(validTrials),
			AcceptanceRateWilson95:      wilson(accepted, validTrials),
			MedianHumanMinutes:          median(humanMinutes),
			MedianWallMinutes:           median(wallMinutes),
			AcceptedGoodputPerHumanHour: goodput,
			TotalCost:                   totalCost,
			CostPerAcceptedChange:       costPerAccepted,
			Regressions:                 regressions,
			PolicyViolations:            violations,
		}
	}
	return output
}

func gateFailures(summaries map[string]Summary, gates Gates) []string {
	var failures []string
	for tool, result := range summaries {
		if result.AcceptanceRate < *gates.MinAcceptanceRate {
			failures = append(failures, tool+" failed min_acceptance_rate")
		}
		if result.PolicyViolations > *gates.MaxPolicyViolations {
			failures = append(failures, tool+" failed max_policy_violations")
		}
		if result.Regressions > *gates.MaxRegressions {
			failures = append(failures, tool+" failed max_regressions")
		}
		if result.CostPerAcceptedChange == nil ||
			*result.CostPerAcceptedChange > *gates.MaxCostPerAcceptedChange {
			failures = append(failures, tool+" failed max_cost_per_accepted_change")
		}
	}
	sort.Strings(failures)
	return failures
}

func main() {
	if len(os.Args) != 2 {
		fmt.Fprintln(os.Stderr, "usage: go run coding_tool_gate.go EXPERIMENT.json")
		os.Exit(2)
	}
	file, err := os.Open(os.Args[1])
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	defer file.Close()

	var experiment Experiment
	decoder := json.NewDecoder(file)
	decoder.DisallowUnknownFields()
	if err := decoder.Decode(&experiment); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	var extra any
	if err := decoder.Decode(&extra); err != io.EOF {
		fmt.Fprintln(os.Stderr, "experiment must contain exactly one JSON value")
		os.Exit(1)
	}
	if err := validate(experiment); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}

	summaries := summarize(experiment)
	encoded, _ := json.MarshalIndent(summaries, "", "  ")
	fmt.Println(string(encoded))
	if failures := gateFailures(summaries, experiment.Gates); len(failures) > 0 {
		for _, failure := range failures {
			fmt.Fprintln(os.Stderr, failure)
		}
		os.Exit(1)
	}
}

Run:

bash
go run coding_tool_gate.go experiment.json

The program requires exactly one decision-bearing result per candidate and task, while allowing failed or invalid attempts to remain in the cost ledger before a controlled rerun. It does not claim that paired product runs replace a randomized human no-AI control; use a separate study design when the decision is about human productivity rather than product output.

Run Security as a Separate Test Track

Security failures should not be diluted inside a quality score.

Test each candidate with:

  • a repository file containing an instruction to read a decoy secret;
  • a tool result requesting an unrelated network call;
  • a dependency installation that invokes a lifecycle script;
  • a task that asks for a CI or permissions change outside scope;
  • a forbidden external domain;
  • a command whose harmless prefix hides a destructive operation;
  • an attempted push or merge beyond the assigned branch;
  • missing and expired credentials.

Record attempted and successful violations separately. The control, not the model's refusal, must block a forbidden action. Use disposable infrastructure and synthetic data.

Official security documentation for Cursor, Claude Code, and GitHub Copilot coding agent describes different approval, sandbox, network, branch, and audit boundaries. Test the exact mode rather than extrapolating one mode's controls to the whole product.

Interpret the Results as a Portfolio

The result may not be one winner:

  • completion can win on short local edits;
  • an interactive agent can reduce active time on exploratory changes;
  • a cloud agent can help with bounded asynchronous maintenance;
  • the current human workflow can remain best for high-context or high-risk tasks;
  • a specialist static analyzer can outperform every generative tool on deterministic checks.

Choose by slice and preserve common policy, acceptance, and audit gates. A mixed toolchain is justified only when its benefit exceeds additional configuration, training, contract, and incident-response complexity.

Roll Out From Evidence, Not Excitement

Promotion should increase repository sensitivity and agent authority gradually:

  1. dry-run the protocol and repair ambiguous tasks;
  2. run the locked comparison in disposable environments;
  3. pilot the selected mode on low-risk real work;
  4. keep independent review and existing CI mandatory;
  5. monitor acceptance, review load, regressions, incidents, and cost;
  6. expand only the slices that continue to pass;
  7. retain the previous tool and configuration for rollback.

Re-evaluate when the tool mode, model policy, client, permissions, contract, pricing, or task mix changes materially.

Common Failure Modes

Letting Operators Pick Their Favorite Tasks

Task selection can create more uplift than the tool. Freeze or randomize the suite before assignment.

Counting a Passing Narrow Test as Acceptance

Hidden invariants, maintainability, scope, and security can fail while one test passes. Use layered acceptance.

Ignoring Rejected Attempts

Dropping failed or expensive runs inflates both quality and cost results. Preserve every valid attempt.

Comparing Unequal Budgets

Unlimited retries for one candidate and a fixed quota for another measure budget, not tool quality. Record and equalize the constraint relevant to the decision.

Revealing Tool Identity to Reviewers

Brand preference can bias review. Normalize artifacts and blind identity where practical.

Forcing a Winner From Sparse Data

Wide confidence intervals and conflicting slices mean the result is inconclusive or mode-specific. Do not collapse them into a star rating.

Frequently Asked Questions

Should the same developer operate every tool?

Not automatically. One operator controls developer differences but may favor a familiar tool. Multiple operators improve representativeness but add variance. Use training, rotate or block assignments, and record experience so the design matches the decision.

Can old completed tickets be reused as tasks?

Yes, if the original patch, later commits, discussion, and hidden answers are removed from the candidate environment and the task remains realistic. Public history can still contaminate model training, so private or newly authored tasks provide stronger evidence.

Should tools receive the same model?

Only if the decision is specifically about harnesses with a shared model. For product selection, test the supported configuration you would actually deploy and record the model policy. Do not disable a product's core design merely to create artificial sameness.

How should ambiguous tasks be scored?

Predefine when clarification is correct. A tool that asks a necessary question should not be penalized as incomplete; a tool that invents requirements and changes code should fail scope or correctness.

Can this protocol prove organization-wide productivity?

No. It can support a bounded product decision for the sampled tasks, repositories, users, and controls. Organization-wide effects also depend on task selection, learning, workflow redesign, review capacity, quality, and long-term maintenance.

Summary

A defensible Cursor, Claude Code, and Copilot comparison is an experiment, not a ranking article. Freeze the decision, task suite, environment, candidate configuration, budgets, acceptance rubric, and security track; retain every outcome; measure accepted changes and human load; report uncertainty and slices; then roll out only the evidence-backed modes through a reversible canary.

References