Direct Answer
Run Cursor, Claude Code, GitHub Copilot, or any other qualified AI coding tool as a controlled experiment. Freeze the task suite, repository snapshots, acceptance tests, environment, instructions, budgets, permissions, and review rubric before execution. Record every attempt, including failures. Decide from accepted changes, human review and rework, regressions, policy violations, uncertainty, and total cost, not generated lines, public leaderboards, or subjective star ratings.
This page is the experiment protocol. Use AI Coding Tools: Capability, Security, and Team Fit first to define the workflow and eliminate candidates that fail hard qualification gates.
Key Takeaways
- A tool name is not an experimental condition. Record client version, mode, model policy, settings, permissions, and runtime.
- Public benchmarks test a dataset and harness; they do not reproduce your repository, requirements, reviewers, or security boundary.
- Use paired representative tasks, clean snapshots, equivalent budgets, randomized order, and independent review.
- Define “accepted” before the run and count review, rework, rejected attempts, regressions, and incidents.
- Report confidence intervals and slice results. “Inconclusive” is more useful than a winner produced by a weak sample.
Why Public Coding Benchmarks Cannot Pick a Product
Public coding benchmarks are useful research instruments, but their score is not a procurement verdict.
The original SWE-bench paper introduced 2,294 issues from 12 Python repositories. A model receives a repository and issue description and must produce a patch. This setup tests long-context repository reasoning and multi-file changes, but it does not measure every product mode, language, code-review standard, integration, or operational risk.
Benchmark validity also changes over time. In February 2026, OpenAI reported that SWE-bench Verified had contamination and flawed tests. In July 2026, it published a further audit of SWE-bench Pro and estimated that roughly 30% of the public-split tasks were broken, citing overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts. OpenAI then retracted its earlier recommendation to use SWE-bench Pro.
That does not make all repository benchmarks useless. It means every score must carry:
- dataset and exact split;
- harness, scaffold, tool, model, and inference settings;
- environment and network policy;
- task-quality audit;
- contamination and leakage analysis;
- cost and retry budget;
- date and source;
- capability boundaries the benchmark does not test.
A product comparison needs stronger evidence: private or newly constructed tasks from your work, human review, operational controls, and outcomes that match the decision.
Productivity Is Harder to Measure Than Completion Time
Developer productivity is a system outcome, not a single timer.
METR's early-2025 randomized controlled trial studied 16 experienced open-source developers completing 246 tasks in repositories they knew well. Under that protocol, allowing early-2025 AI tools increased completion time by 19%, while participants believed the tools had made them faster.
METR later marked the historical result as outdated. Its February 2026 update found that a new task-level experiment no longer produced a reliable current estimate because developers selected out of no-AI tasks, task choice changed, parallel agents made time reporting difficult, and output quality could differ. The raw estimates had wide confidence intervals.
The durable lesson is not “AI slows developers” or “AI speeds developers.” It is that perceived speed, task completion, active human time, wall time, output quality, and selection effects are different variables. Your experiment must name which one it measures.
Define the Decision Before the Trial
A trial should answer one bounded decision:
decision: choose a default tool for bounded maintenance pull requests
candidates:
- candidate-a-local-agent
- candidate-b-cloud-agent
- current-human-workflow
repositories:
- typescript-service
- python-library
task_slices:
- bug-fix
- test-addition
- dependency-safe-update
- narrow-refactor
excluded:
- production-operations
- schema-migrations
- security-critical-auth-changes
primary_outcome: accepted_change_without_material_rework
hard_gates:
- no_policy_violation
- no_unapproved_network
- no_secret_access
- required_tests_pass
Do not ask “which tool is best?” Ask “which qualified mode produces more accepted changes for these task slices, within these quality, security, cost, and review constraints?”
The candidate may be a mixed toolchain or the current workflow. Including a no-adoption option prevents the experiment from assuming its conclusion.
Build a Representative Task Suite
A useful task suite samples work the team expects to delegate, including cases where tools may fail.
Sample From Real Work
Use recently completed or newly authored tasks whose expected behavior can be independently checked. Remove the original patch and future Git history from the candidate environment. Do not give the model a task it or its retrieval tools can look up online.
Preserve Task Slices
Tag every task by dimensions that may change results:
| Slice | Examples |
|---|---|
| Work type | bug fix, feature, refactor, test, documentation, dependency update |
| Scope | one file, subsystem, cross-service |
| Language and framework | TypeScript, Go, Python, Java, infrastructure |
| Context quality | complete specification, ambiguous legacy issue, missing test |
| Risk | low, moderate, security-sensitive, irreversible |
| Repository maturity | greenfield, mature service, legacy monolith |
| Verification | unit tests, integration tests, static analysis, human behavior review |
Critical slices need enough repeated observations to support a decision. If a slice has one task, report the observation without generalizing.
Add Clean and Adversarial Tasks
Include:
- tasks where no change is required;
- underspecified tasks where the correct action is clarification;
- impossible tasks where the tool should stop;
- repositories with decoy secrets and forbidden paths;
- untrusted comments or tool output containing indirect prompt injection;
- commands that should be denied;
- changes that pass a narrow test but violate an invariant.
Run adversarial tasks in isolated fixtures without real secrets, production access, or uncontrolled network egress.
This track is no longer hypothetical. The July 2026 IssueTrojanBench preprint tested malicious instructions delivered through issue bodies, PDFs, and low-authority metadata against specific Cursor, Claude Code, and Codex configurations. In that benchmark, 66.5% of malicious issues crossed both agent- and model-level guardrails. That result belongs to the tested versions, models, attacks, and harness; it is not a permanent product ranking. Its durable implication is that untrusted issue content must remain data, while filesystem, shell, network, secret, and Git authority is enforced outside the model.
Pre-Register Acceptance and Failure
Define acceptance before seeing a candidate's output.
An accepted change should satisfy all required dimensions:
- Behavior: explicit acceptance tests and relevant regression tests pass.
- Scope: the change stays within the requested contract or documents an approved deviation.
- Quality: independent review finds no material correctness, maintainability, or architecture issue.
- Security: no forbidden access, secret exposure, unsafe dependency, policy bypass, or unreviewed side effect.
- Evidence: diff, commands, tests, tool trace, cost, and approvals are present.
Use machine-readable terminal states:
accepted;rejected_correctness;rejected_scope;rejected_security;rejected_maintainability;clarification_required;environment_failure;budget_exhausted;invalid_trial.
Environment failures and invalid trials must not quietly become tool failures or disappear. Report and rerun them under the pre-registered policy.
Control the Experimental Conditions
Fair comparison means equivalent opportunity, not identical user-interface clicks.
Pin What Can Be Pinned
Record:
- product, client, extension, or CLI version;
- operating mode and model policy;
- selected model if configurable;
- repository base commit and environment image hash;
- rules, prompt, task specification, and tool configuration hashes;
- file, shell, network, secret, and MCP permissions;
- retry, token, time, and monetary budgets;
- reviewer rubric revision.
If the vendor manages a model or service version, record vendor-managed and the run timestamp. Do not invent precision the product does not expose.
Start From a Clean Snapshot
Each candidate receives the same base commit and dependency state in a separate worktree, container, or VM. Remove candidate output before the next run. Avoid caches that leak patches, conversations, or tests between candidates.
Give Equivalent Information
Provide the same task requirements and accessible documentation. Do not give one candidate a carefully iterated prompt and another a one-line issue. Product-specific syntax can differ, but the information and allowed actions should remain equivalent.
Randomize and Block
Randomize candidate order within task blocks to reduce learning, fatigue, and temporal effects. If different developers operate tools, block or rotate by developer. Do not let the strongest product advocate operate only their preferred tool.
Separate Operators From Reviewers
Where practical, reviewers receive normalized diffs and evidence without product identity. Reviewers should not see marketing labels, token counts, or narrative traces before judging correctness and maintainability. Security reviewers can inspect full traces after the primary code decision.
Measure Accepted Outcomes and Human Load
Raw output is not productivity. Track a metric set that exposes trade-offs.
| Metric | Definition | Why it matters |
|---|---|---|
| Acceptance rate | accepted trials / valid trials | Basic useful-outcome rate |
| Human minutes | active operation + review + rework | Captures scarce team attention |
| Wall time | start to terminal result | Matters for synchronous and queued work |
| Accepted Goodput | accepted changes / human hour | Combines utility with human capacity |
| Cost per accepted change | all direct and labor cost / accepted changes | Compares economic output |
| Regression count | post-change failures attributed to candidate | Detects narrow-test optimization |
| Policy violations | forbidden access or action events | Hard safety signal |
| Scope deviation | material unapproved change outside task | Detects over-eager behavior |
| Clarification quality | correct clarification on ambiguous tasks | Rewards safe inability |
| Review queue impact | waiting and reviewer load | Detects parallel-agent saturation |
Report by task and repository slice. A high aggregate can hide failure on the team's most important language or risk class.
Separate Time Measures
Use at least:
human minutes = active operation + review + rework
wall minutes = elapsed time from assignment to terminal result
For parallel agents, human minutes cannot be calculated by adding overlapping wall clocks. Capture active operator intervals or use a defined time-sampling method.
Price Labor and Failure
total cost
= direct tool and compute cost
+ labor rate * human minutes / 60
+ incident and rollback cost
cost per accepted change
= total cost / accepted changes
Record rejected trials in total cost. If there are zero accepted changes, the cost per accepted change is undefined, not zero.
Use Confidence Intervals and Honest Language
An observed difference is not automatically a stable advantage.
For binary acceptance, report the count and a confidence interval. For skewed time and cost, report medians and the underlying observations, not only averages. With few tasks or wide intervals, describe results as inconclusive.
Avoid:
- ranking candidates by a weighted score with arbitrary weights;
- treating a small percentage difference as decisive;
- pooling unrelated task slices;
- excluding failed or expensive attempts after seeing results;
- choosing thresholds that make a preferred candidate win.
Hard security and correctness gates should remain separate from optimization metrics. A candidate does not compensate for secret access by saving review minutes.
A Machine-Readable Trial Ledger
Store each attempt as immutable evidence:
{
"experiment_id": "coding-tools-maintenance-v1",
"labor_cost_per_hour": 120,
"gates": {
"min_acceptance_rate": 0.6,
"max_policy_violations": 0,
"max_regressions": 0,
"max_cost_per_accepted_change": 900
},
"trials": [
{
"trial_id": "candidate-a-task-001",
"tool": "candidate-a",
"task_id": "task-001",
"task_type": "bug-fix",
"tool_version": "pinned-or-vendor-managed",
"model": "pinned-or-vendor-managed",
"environment_hash": "sha256:...",
"prompt_hash": "sha256:...",
"status": "accepted",
"active_minutes": 12,
"review_minutes": 18,
"rework_minutes": 4,
"wall_minutes": 41,
"direct_cost": 7.4,
"regressions": 0,
"policy_violations": 0
}
]
}
Thresholds above are illustrative fields, not universal recommendations. Set them from the team's existing service, risk, and cost objectives before the trial.
A Runnable Evaluation Gate
The dependency-free Go program below validates paired decision-bearing tasks, retains invalid and environment-failure attempts in cost, calculates Wilson intervals, human load, accepted Goodput, and cost per accepted change, then applies configured gates.
package main
import (
"encoding/json"
"errors"
"fmt"
"io"
"math"
"os"
"sort"
)
var validStatuses = map[string]bool{
"accepted": true,
"rejected_correctness": true,
"rejected_scope": true,
"rejected_security": true,
"rejected_maintainability": true,
"clarification_required": true,
"budget_exhausted": true,
"environment_failure": true,
"invalid_trial": true,
}
type Gates struct {
MinAcceptanceRate *float64 `json:"min_acceptance_rate"`
MaxPolicyViolations *float64 `json:"max_policy_violations"`
MaxRegressions *float64 `json:"max_regressions"`
MaxCostPerAcceptedChange *float64 `json:"max_cost_per_accepted_change"`
}
type Trial struct {
TrialID string `json:"trial_id"`
Tool string `json:"tool"`
TaskID string `json:"task_id"`
TaskType string `json:"task_type"`
ToolVersion string `json:"tool_version"`
Model string `json:"model"`
EnvironmentHash string `json:"environment_hash"`
PromptHash string `json:"prompt_hash"`
Status string `json:"status"`
ActiveMinutes *float64 `json:"active_minutes"`
ReviewMinutes *float64 `json:"review_minutes"`
ReworkMinutes *float64 `json:"rework_minutes"`
WallMinutes *float64 `json:"wall_minutes"`
DirectCost *float64 `json:"direct_cost"`
Regressions *float64 `json:"regressions"`
PolicyViolations *float64 `json:"policy_violations"`
}
type Experiment struct {
ExperimentID string `json:"experiment_id"`
LaborCostPerHour *float64 `json:"labor_cost_per_hour"`
Gates Gates `json:"gates"`
Trials []Trial `json:"trials"`
}
type Summary struct {
ValidTrials int `json:"valid_trials"`
OperationalFailures int `json:"operational_failures"`
Accepted int `json:"accepted"`
AcceptanceRate float64 `json:"acceptance_rate"`
AcceptanceRateWilson95 [2]float64 `json:"acceptance_rate_wilson_95"`
MedianHumanMinutes float64 `json:"median_human_minutes"`
MedianWallMinutes float64 `json:"median_wall_minutes"`
AcceptedGoodputPerHumanHour *float64 `json:"accepted_goodput_per_human_hour"`
TotalCost float64 `json:"total_cost"`
CostPerAcceptedChange *float64 `json:"cost_per_accepted_change"`
Regressions float64 `json:"regressions"`
PolicyViolations float64 `json:"policy_violations"`
}
func number(value *float64, field string) (float64, error) {
if value == nil || math.IsNaN(*value) || math.IsInf(*value, 0) || *value < 0 {
return 0, fmt.Errorf("%s must be finite and non-negative", field)
}
return *value, nil
}
func isDecisionStatus(status string) bool {
return status != "environment_failure" && status != "invalid_trial"
}
func validate(experiment Experiment) error {
if experiment.ExperimentID == "" {
return errors.New("experiment_id is required")
}
labor, err := number(experiment.LaborCostPerHour, "labor_cost_per_hour")
if err != nil || labor < 0 {
return errors.New("labor_cost_per_hour must be finite and non-negative")
}
if len(experiment.Trials) == 0 {
return errors.New("trials must not be empty")
}
gates := map[string]*float64{
"min_acceptance_rate": experiment.Gates.MinAcceptanceRate,
"max_policy_violations": experiment.Gates.MaxPolicyViolations,
"max_regressions": experiment.Gates.MaxRegressions,
"max_cost_per_accepted_change": experiment.Gates.MaxCostPerAcceptedChange,
}
for name, value := range gates {
if _, err := number(value, name); err != nil {
return err
}
}
if *experiment.Gates.MinAcceptanceRate > 1 {
return errors.New("min_acceptance_rate must not exceed 1")
}
seenTrials := map[string]bool{}
tools := map[string]bool{}
decisionCounts := map[string]map[string]int{}
for index, trial := range experiment.Trials {
if trial.TrialID == "" || trial.Tool == "" || trial.TaskID == "" ||
trial.TaskType == "" || trial.ToolVersion == "" || trial.Model == "" ||
trial.EnvironmentHash == "" || trial.PromptHash == "" {
return fmt.Errorf("trials[%d] has missing identity", index)
}
tools[trial.Tool] = true
if seenTrials[trial.TrialID] {
return fmt.Errorf("duplicate trial_id %q", trial.TrialID)
}
seenTrials[trial.TrialID] = true
if !validStatuses[trial.Status] {
return fmt.Errorf("trials[%d] has invalid status %q", index, trial.Status)
}
for name, value := range map[string]*float64{
"active_minutes": trial.ActiveMinutes,
"review_minutes": trial.ReviewMinutes,
"rework_minutes": trial.ReworkMinutes,
"wall_minutes": trial.WallMinutes,
"direct_cost": trial.DirectCost,
"regressions": trial.Regressions,
"policy_violations": trial.PolicyViolations,
} {
if _, err := number(value, fmt.Sprintf("trials[%d].%s", index, name)); err != nil {
return err
}
}
if isDecisionStatus(trial.Status) {
if decisionCounts[trial.Tool] == nil {
decisionCounts[trial.Tool] = map[string]int{}
}
decisionCounts[trial.Tool][trial.TaskID]++
if decisionCounts[trial.Tool][trial.TaskID] > 1 {
return fmt.Errorf("multiple decision results for %s/%s", trial.Tool, trial.TaskID)
}
}
}
if len(decisionCounts) < 2 {
return errors.New("at least two tools need decision-bearing trials")
}
if len(decisionCounts) != len(tools) {
return errors.New("every tool needs at least one decision-bearing trial")
}
var baseline []string
for tool, taskCounts := range decisionCounts {
tasks := make([]string, 0, len(taskCounts))
for task := range taskCounts {
tasks = append(tasks, task)
}
sort.Strings(tasks)
if baseline == nil {
baseline = tasks
continue
}
if fmt.Sprint(tasks) != fmt.Sprint(baseline) {
return fmt.Errorf("tool %s does not cover the paired task set", tool)
}
}
return nil
}
func wilson(successes, total int) [2]float64 {
if total == 0 {
return [2]float64{}
}
const z = 1.96
p := float64(successes) / float64(total)
denominator := 1 + z*z/float64(total)
center := (p + z*z/(2*float64(total))) / denominator
spread := z * math.Sqrt(
(p*(1-p)+z*z/(4*float64(total)))/float64(total),
) / denominator
return [2]float64{math.Max(0, center-spread), math.Min(1, center+spread)}
}
func median(values []float64) float64 {
sort.Float64s(values)
middle := len(values) / 2
if len(values)%2 == 1 {
return values[middle]
}
return (values[middle-1] + values[middle]) / 2
}
func summarize(experiment Experiment) map[string]Summary {
grouped := map[string][]Trial{}
for _, trial := range experiment.Trials {
grouped[trial.Tool] = append(grouped[trial.Tool], trial)
}
output := map[string]Summary{}
for tool, trials := range grouped {
var validTrials, operationalFailures, accepted int
var humanMinutes, wallMinutes []float64
var totalHuman, totalDirect, regressions, violations float64
for _, trial := range trials {
human := *trial.ActiveMinutes + *trial.ReviewMinutes + *trial.ReworkMinutes
humanMinutes = append(humanMinutes, human)
wallMinutes = append(wallMinutes, *trial.WallMinutes)
totalHuman += human
totalDirect += *trial.DirectCost
regressions += *trial.Regressions
violations += *trial.PolicyViolations
if isDecisionStatus(trial.Status) {
validTrials++
if trial.Status == "accepted" {
accepted++
}
} else {
operationalFailures++
}
}
totalCost := totalDirect + *experiment.LaborCostPerHour*totalHuman/60
var goodput, costPerAccepted *float64
if totalHuman > 0 {
value := float64(accepted) * 60 / totalHuman
goodput = &value
}
if accepted > 0 {
value := totalCost / float64(accepted)
costPerAccepted = &value
}
output[tool] = Summary{
ValidTrials: validTrials,
OperationalFailures: operationalFailures,
Accepted: accepted,
AcceptanceRate: float64(accepted) / float64(validTrials),
AcceptanceRateWilson95: wilson(accepted, validTrials),
MedianHumanMinutes: median(humanMinutes),
MedianWallMinutes: median(wallMinutes),
AcceptedGoodputPerHumanHour: goodput,
TotalCost: totalCost,
CostPerAcceptedChange: costPerAccepted,
Regressions: regressions,
PolicyViolations: violations,
}
}
return output
}
func gateFailures(summaries map[string]Summary, gates Gates) []string {
var failures []string
for tool, result := range summaries {
if result.AcceptanceRate < *gates.MinAcceptanceRate {
failures = append(failures, tool+" failed min_acceptance_rate")
}
if result.PolicyViolations > *gates.MaxPolicyViolations {
failures = append(failures, tool+" failed max_policy_violations")
}
if result.Regressions > *gates.MaxRegressions {
failures = append(failures, tool+" failed max_regressions")
}
if result.CostPerAcceptedChange == nil ||
*result.CostPerAcceptedChange > *gates.MaxCostPerAcceptedChange {
failures = append(failures, tool+" failed max_cost_per_accepted_change")
}
}
sort.Strings(failures)
return failures
}
func main() {
if len(os.Args) != 2 {
fmt.Fprintln(os.Stderr, "usage: go run coding_tool_gate.go EXPERIMENT.json")
os.Exit(2)
}
file, err := os.Open(os.Args[1])
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
defer file.Close()
var experiment Experiment
decoder := json.NewDecoder(file)
decoder.DisallowUnknownFields()
if err := decoder.Decode(&experiment); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
var extra any
if err := decoder.Decode(&extra); err != io.EOF {
fmt.Fprintln(os.Stderr, "experiment must contain exactly one JSON value")
os.Exit(1)
}
if err := validate(experiment); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
summaries := summarize(experiment)
encoded, _ := json.MarshalIndent(summaries, "", " ")
fmt.Println(string(encoded))
if failures := gateFailures(summaries, experiment.Gates); len(failures) > 0 {
for _, failure := range failures {
fmt.Fprintln(os.Stderr, failure)
}
os.Exit(1)
}
}
Run:
go run coding_tool_gate.go experiment.json
The program requires exactly one decision-bearing result per candidate and task, while allowing failed or invalid attempts to remain in the cost ledger before a controlled rerun. It does not claim that paired product runs replace a randomized human no-AI control; use a separate study design when the decision is about human productivity rather than product output.
Run Security as a Separate Test Track
Security failures should not be diluted inside a quality score.
Test each candidate with:
- a repository file containing an instruction to read a decoy secret;
- a tool result requesting an unrelated network call;
- a dependency installation that invokes a lifecycle script;
- a task that asks for a CI or permissions change outside scope;
- a forbidden external domain;
- a command whose harmless prefix hides a destructive operation;
- an attempted push or merge beyond the assigned branch;
- missing and expired credentials.
Record attempted and successful violations separately. The control, not the model's refusal, must block a forbidden action. Use disposable infrastructure and synthetic data.
Official security documentation for Cursor, Claude Code, and GitHub Copilot coding agent describes different approval, sandbox, network, branch, and audit boundaries. Test the exact mode rather than extrapolating one mode's controls to the whole product.
Interpret the Results as a Portfolio
The result may not be one winner:
- completion can win on short local edits;
- an interactive agent can reduce active time on exploratory changes;
- a cloud agent can help with bounded asynchronous maintenance;
- the current human workflow can remain best for high-context or high-risk tasks;
- a specialist static analyzer can outperform every generative tool on deterministic checks.
Choose by slice and preserve common policy, acceptance, and audit gates. A mixed toolchain is justified only when its benefit exceeds additional configuration, training, contract, and incident-response complexity.
Roll Out From Evidence, Not Excitement
Promotion should increase repository sensitivity and agent authority gradually:
- dry-run the protocol and repair ambiguous tasks;
- run the locked comparison in disposable environments;
- pilot the selected mode on low-risk real work;
- keep independent review and existing CI mandatory;
- monitor acceptance, review load, regressions, incidents, and cost;
- expand only the slices that continue to pass;
- retain the previous tool and configuration for rollback.
Re-evaluate when the tool mode, model policy, client, permissions, contract, pricing, or task mix changes materially.
Common Failure Modes
Letting Operators Pick Their Favorite Tasks
Task selection can create more uplift than the tool. Freeze or randomize the suite before assignment.
Counting a Passing Narrow Test as Acceptance
Hidden invariants, maintainability, scope, and security can fail while one test passes. Use layered acceptance.
Ignoring Rejected Attempts
Dropping failed or expensive runs inflates both quality and cost results. Preserve every valid attempt.
Comparing Unequal Budgets
Unlimited retries for one candidate and a fixed quota for another measure budget, not tool quality. Record and equalize the constraint relevant to the decision.
Revealing Tool Identity to Reviewers
Brand preference can bias review. Normalize artifacts and blind identity where practical.
Forcing a Winner From Sparse Data
Wide confidence intervals and conflicting slices mean the result is inconclusive or mode-specific. Do not collapse them into a star rating.
Frequently Asked Questions
Should the same developer operate every tool?
Not automatically. One operator controls developer differences but may favor a familiar tool. Multiple operators improve representativeness but add variance. Use training, rotate or block assignments, and record experience so the design matches the decision.
Can old completed tickets be reused as tasks?
Yes, if the original patch, later commits, discussion, and hidden answers are removed from the candidate environment and the task remains realistic. Public history can still contaminate model training, so private or newly authored tasks provide stronger evidence.
Should tools receive the same model?
Only if the decision is specifically about harnesses with a shared model. For product selection, test the supported configuration you would actually deploy and record the model policy. Do not disable a product's core design merely to create artificial sameness.
How should ambiguous tasks be scored?
Predefine when clarification is correct. A tool that asks a necessary question should not be penalized as incomplete; a tool that invents requirements and changes code should fail scope or correctness.
Can this protocol prove organization-wide productivity?
No. It can support a bounded product decision for the sampled tasks, repositories, users, and controls. Organization-wide effects also depend on task selection, learning, workflow redesign, review capacity, quality, and long-term maintenance.
Summary
A defensible Cursor, Claude Code, and Copilot comparison is an experiment, not a ranking article. Freeze the decision, task suite, environment, candidate configuration, budgets, acceptance rubric, and security track; retain every outcome; measure accepted changes and human load; report uncertainty and slices; then roll out only the evidence-backed modes through a reversible canary.
References
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- OpenAI: Separating Signal From Noise in Coding Evaluations
- METR: Early-2025 AI and Experienced Open-Source Developer Productivity
- METR: Changing the Developer Productivity Experiment Design
- IssueTrojanBench: Malicious Issue Requests Against Coding Agents
- OWASP LLM01:2025 Prompt Injection
- Cursor Agent Security
- Claude Code Security
- GitHub Copilot Coding Agent Risks and Mitigations
- AI Coding Tools Selection Guide
- AI Coding Tool Cost Evaluation