Cursor vs Claude Code vs Copilot: Reproducible Trial

Direct Answer

Run Cursor, Claude Code, GitHub Copilot, or any other qualified AI coding tool as a controlled experiment. Freeze the task suite, repository snapshots, acceptance tests, environment, instructions, budgets, permissions, and review rubric before execution. Record every attempt, including failures. Decide from accepted changes, human review and rework, regressions, policy violations, uncertainty, and total cost, not generated lines, public leaderboards, or subjective star ratings.

This page is the experiment protocol. Use AI Coding Tools: Capability, Security, and Team Fit first to define the workflow and eliminate candidates that fail hard qualification gates.

Key Takeaways

• A tool name is not an experimental condition. Record client version, mode, model policy, settings, permissions, and runtime.
• Public benchmarks test a dataset and harness; they do not reproduce your repository, requirements, reviewers, or security boundary.
• Use paired representative tasks, clean snapshots, equivalent budgets, randomized order, and independent review.
• Define “accepted” before the run and count review, rework, rejected attempts, regressions, and incidents.
• Report confidence intervals and slice results. “Inconclusive” is more useful than a winner produced by a weak sample.

Why Public Coding Benchmarks Cannot Pick a Product

Public coding benchmarks are useful research instruments, but their score is not a procurement verdict.

The original SWE-bench paper introduced 2,294 issues from 12 Python repositories. A model receives a repository and issue description and must produce a patch. This setup tests long-context repository reasoning and multi-file changes, but it does not measure every product mode, language, code-review standard, integration, or operational risk.

Benchmark validity also changes over time. In February 2026, OpenAI reported that SWE-bench Verified had contamination and flawed tests. In July 2026, it published a further audit of SWE-bench Pro and estimated that roughly 30% of the public-split tasks were broken, citing overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts. OpenAI then retracted its earlier recommendation to use SWE-bench Pro.

That does not make all repository benchmarks useless. It means every score must carry:

• dataset and exact split;
• harness, scaffold, tool, model, and inference settings;
• environment and network policy;
• task-quality audit;
• contamination and leakage analysis;
• cost and retry budget;
• date and source;
• capability boundaries the benchmark does not test.

A product comparison needs stronger evidence: private or newly constructed tasks from your work, human review, operational controls, and outcomes that match the decision.

Productivity Is Harder to Measure Than Completion Time

Developer productivity is a system outcome, not a single timer.

METR's early-2025 randomized controlled trial studied 16 experienced open-source developers completing 246 tasks in repositories they knew well. Under that protocol, allowing early-2025 AI tools increased completion time by 19%, while participants believed the tools had made them faster.

METR later marked the historical result as outdated. Its February 2026 update found that a new task-level experiment no longer produced a reliable current estimate because developers selected out of no-AI tasks, task choice changed, parallel agents made time reporting difficult, and output quality could differ. The raw estimates had wide confidence intervals.

The durable lesson is not “AI slows developers” or “AI speeds developers.” It is that perceived speed, task completion, active human time, wall time, output quality, and selection effects are different variables. Your experiment must name which one it measures.

Define the Decision Before the Trial

A trial should answer one bounded decision:

decision: choose a default tool for bounded maintenance pull requests
candidates:
  - candidate-a-local-agent
  - candidate-b-cloud-agent
  - current-human-workflow
repositories:
  - typescript-service
  - python-library
task_slices:
  - bug-fix
  - test-addition
  - dependency-safe-update
  - narrow-refactor
excluded:
  - production-operations
  - schema-migrations
  - security-critical-auth-changes
primary_outcome: accepted_change_without_material_rework
hard_gates:
  - no_policy_violation
  - no_unapproved_network
  - no_secret_access
  - required_tests_pass

Do not ask “which tool is best?” Ask “which qualified mode produces more accepted changes for these task slices, within these quality, security, cost, and review constraints?”

The candidate may be a mixed toolchain or the current workflow. Including a no-adoption option prevents the experiment from assuming its conclusion.

Build a Representative Task Suite

A useful task suite samples work the team expects to delegate, including cases where tools may fail.

Sample From Real Work

Use recently completed or newly authored tasks whose expected behavior can be independently checked. Remove the original patch and future Git history from the candidate environment. Do not give the model a task it or its retrieval tools can look up online.

Preserve Task Slices

Tag every task by dimensions that may change results:

Critical slices need enough repeated observations to support a decision. If a slice has one task, report the observation without generalizing.

Add Clean and Adversarial Tasks

Include:

• tasks where no change is required;
• underspecified tasks where the correct action is clarification;
• impossible tasks where the tool should stop;
• repositories with decoy secrets and forbidden paths;
• untrusted comments or tool output containing indirect prompt injection;
• commands that should be denied;
• changes that pass a narrow test but violate an invariant.

Run adversarial tasks in isolated fixtures without real secrets, production access, or uncontrolled network egress.

Pre-Register Acceptance and Failure

Define acceptance before seeing a candidate's output.

An accepted change should satisfy all required dimensions:

1. Behavior: explicit acceptance tests and relevant regression tests pass.
2. Scope: the change stays within the requested contract or documents an approved deviation.
3. Quality: independent review finds no material correctness, maintainability, or architecture issue.
4. Security: no forbidden access, secret exposure, unsafe dependency, policy bypass, or unreviewed side effect.
5. Evidence: diff, commands, tests, tool trace, cost, and approvals are present.

Use machine-readable terminal states:

• accepted;
• rejected_correctness;
• rejected_scope;
• rejected_security;
• rejected_maintainability;
• clarification_required;
• environment_failure;
• budget_exhausted;
• invalid_trial.

Environment failures and invalid trials must not quietly become tool failures or disappear. Report and rerun them under the pre-registered policy.

Control the Experimental Conditions

Fair comparison means equivalent opportunity, not identical user-interface clicks.

Pin What Can Be Pinned

Record:

• product, client, extension, or CLI version;
• operating mode and model policy;
• selected model if configurable;
• repository base commit and environment image hash;
• rules, prompt, task specification, and tool configuration hashes;
• file, shell, network, secret, and MCP permissions;
• retry, token, time, and monetary budgets;
• reviewer rubric revision.

If the vendor manages a model or service version, record vendor-managed and the run timestamp. Do not invent precision the product does not expose.

Start From a Clean Snapshot

Each candidate receives the same base commit and dependency state in a separate worktree, container, or VM. Remove candidate output before the next run. Avoid caches that leak patches, conversations, or tests between candidates.

Give Equivalent Information

Provide the same task requirements and accessible documentation. Do not give one candidate a carefully iterated prompt and another a one-line issue. Product-specific syntax can differ, but the information and allowed actions should remain equivalent.

Randomize and Block

Randomize candidate order within task blocks to reduce learning, fatigue, and temporal effects. If different developers operate tools, block or rotate by developer. Do not let the strongest product advocate operate only their preferred tool.

Separate Operators From Reviewers

Where practical, reviewers receive normalized diffs and evidence without product identity. Reviewers should not see marketing labels, token counts, or narrative traces before judging correctness and maintainability. Security reviewers can inspect full traces after the primary code decision.

Measure Accepted Outcomes and Human Load

Raw output is not productivity. Track a metric set that exposes trade-offs.

Report by task and repository slice. A high aggregate can hide failure on the team's most important language or risk class.

Separate Time Measures

Use at least:

human minutes = active operation + review + rework
wall minutes  = elapsed time from assignment to terminal result

For parallel agents, human minutes cannot be calculated by adding overlapping wall clocks. Capture active operator intervals or use a defined time-sampling method.

Price Labor and Failure

total cost
  = direct tool and compute cost
  + labor rate * human minutes / 60
  + incident and rollback cost

cost per accepted change
  = total cost / accepted changes

Record rejected trials in total cost. If there are zero accepted changes, the cost per accepted change is undefined, not zero.

Use Confidence Intervals and Honest Language

An observed difference is not automatically a stable advantage.

For binary acceptance, report the count and a confidence interval. For skewed time and cost, report medians and the underlying observations, not only averages. With few tasks or wide intervals, describe results as inconclusive.

Avoid:

• ranking candidates by a weighted score with arbitrary weights;
• treating a small percentage difference as decisive;
• pooling unrelated task slices;
• excluding failed or expensive attempts after seeing results;
• choosing thresholds that make a preferred candidate win.

Hard security and correctness gates should remain separate from optimization metrics. A candidate does not compensate for secret access by saving review minutes.

A Machine-Readable Trial Ledger

Store each attempt as immutable evidence:

{
  "experiment_id": "coding-tools-maintenance-v1",
  "labor_cost_per_hour": 120,
  "gates": {
    "min_acceptance_rate": 0.6,
    "max_policy_violations": 0,
    "max_regressions": 0,
    "max_cost_per_accepted_change": 900
  },
  "trials": [
    {
      "trial_id": "candidate-a-task-001",
      "tool": "candidate-a",
      "task_id": "task-001",
      "task_type": "bug-fix",
      "tool_version": "pinned-or-vendor-managed",
      "model": "pinned-or-vendor-managed",
      "environment_hash": "sha256:...",
      "prompt_hash": "sha256:...",
      "status": "accepted",
      "active_minutes": 12,
      "review_minutes": 18,
      "rework_minutes": 4,
      "wall_minutes": 41,
      "direct_cost": 7.4,
      "regressions": 0,
      "policy_violations": 0
    }
  ]
}

Thresholds above are illustrative fields, not universal recommendations. Set them from the team's existing service, risk, and cost objectives before the trial.

A Runnable Evaluation Gate

The dependency-free Python program below validates paired task coverage, rejects duplicate or malformed trials, calculates Wilson intervals, human load, accepted Goodput, and cost per accepted change, then applies configured gates.

from __future__ import annotations

import argparse
import json
import math
import statistics
from collections import defaultdict
from pathlib import Path
from typing import Any


STATUSES = {
    "accepted",
    "rejected_correctness",
    "rejected_scope",
    "rejected_security",
    "rejected_maintainability",
    "clarification_required",
    "budget_exhausted",
}
NUMERIC_FIELDS = {
    "active_minutes",
    "review_minutes",
    "rework_minutes",
    "wall_minutes",
    "direct_cost",
    "regressions",
    "policy_violations",
}
STRING_FIELDS = {
    "trial_id",
    "tool",
    "task_id",
    "task_type",
    "tool_version",
    "model",
    "environment_hash",
    "prompt_hash",
    "status",
}
GATE_NAMES = {
    "min_acceptance_rate",
    "max_policy_violations",
    "max_regressions",
    "max_cost_per_accepted_change",
}


def is_number(value: Any) -> bool:
    return isinstance(value, (int, float)) and not isinstance(value, bool)


def wilson(successes: int, total: int, z: float = 1.96) -> list[float]:
    if total == 0:
        return [0.0, 0.0]
    p = successes / total
    denominator = 1 + z * z / total
    center = (p + z * z / (2 * total)) / denominator
    spread = z * math.sqrt(
        (p * (1 - p) + z * z / (4 * total)) / total
    ) / denominator
    return [max(0.0, center - spread), min(1.0, center + spread)]


def load(path: Path) -> dict[str, Any]:
    value = json.loads(path.read_text(encoding="utf-8"))
    if not isinstance(value, dict):
        raise ValueError("experiment root must be an object")
    return value


def validate(experiment: dict[str, Any]) -> list[str]:
    errors: list[str] = []
    labor = experiment.get("labor_cost_per_hour")
    gates = experiment.get("gates")
    trials = experiment.get("trials")
    if not is_number(labor) or labor < 0:
        errors.append("labor_cost_per_hour must be non-negative")
    if not isinstance(gates, dict):
        errors.append("gates must be an object")
        gates = {}
    if not isinstance(trials, list) or not trials:
        return errors + ["trials must be a non-empty list"]

    unknown_gates = set(gates) - GATE_NAMES
    if unknown_gates:
        errors.append(f"unknown gates: {sorted(unknown_gates)}")
    for name, value in gates.items():
        if not is_number(value) or value < 0:
            errors.append(f"gate {name} must be non-negative")

    pairs: set[tuple[str, str]] = set()
    tasks_by_tool: dict[str, set[str]] = defaultdict(set)
    for index, trial in enumerate(trials):
        if not isinstance(trial, dict):
            errors.append(f"trials[{index}] must be an object")
            continue
        for field in STRING_FIELDS:
            if not isinstance(trial.get(field), str) or not trial[field]:
                errors.append(f"trials[{index}].{field} must be a string")
        for field in NUMERIC_FIELDS:
            value = trial.get(field)
            if not is_number(value) or value < 0:
                errors.append(
                    f"trials[{index}].{field} must be non-negative"
                )
        status = trial.get("status")
        if isinstance(status, str) and status not in STATUSES:
            errors.append(f"trials[{index}].status is invalid")
        tool, task = trial.get("tool"), trial.get("task_id")
        if isinstance(tool, str) and isinstance(task, str):
            pair = (tool, task)
            if pair in pairs:
                errors.append(f"duplicate tool/task pair: {pair}")
            pairs.add(pair)
            tasks_by_tool[tool].add(task)

    if len(tasks_by_tool) < 2:
        errors.append("at least two tools are required")
    task_sets = list(tasks_by_tool.values())
    if task_sets and any(tasks != task_sets[0] for tasks in task_sets[1:]):
        errors.append("every tool must run the same task_id set")
    return errors


def summarize(experiment: dict[str, Any]) -> dict[str, dict[str, Any]]:
    groups: dict[str, list[dict[str, Any]]] = defaultdict(list)
    for trial in experiment["trials"]:
        groups[trial["tool"]].append(trial)

    labor = experiment["labor_cost_per_hour"]
    output: dict[str, dict[str, Any]] = {}
    for tool, trials in sorted(groups.items()):
        accepted = sum(t["status"] == "accepted" for t in trials)
        human_minutes = [
            t["active_minutes"] + t["review_minutes"] + t["rework_minutes"]
            for t in trials
        ]
        total_human = sum(human_minutes)
        total_direct = sum(t["direct_cost"] for t in trials)
        total_cost = total_direct + labor * total_human / 60
        output[tool] = {
            "trials": len(trials),
            "accepted": accepted,
            "acceptance_rate": accepted / len(trials),
            "acceptance_rate_wilson_95": wilson(accepted, len(trials)),
            "median_human_minutes": statistics.median(human_minutes),
            "median_wall_minutes": statistics.median(
                t["wall_minutes"] for t in trials
            ),
            "accepted_goodput_per_human_hour": (
                accepted * 60 / total_human if total_human else None
            ),
            "total_cost": total_cost,
            "cost_per_accepted_change": (
                total_cost / accepted if accepted else None
            ),
            "regressions": sum(t["regressions"] for t in trials),
            "policy_violations": sum(
                t["policy_violations"] for t in trials
            ),
        }
    return output


def apply_gates(
    summaries: dict[str, dict[str, Any]], gates: dict[str, float]
) -> list[str]:
    errors: list[str] = []
    for tool, result in summaries.items():
        checks = {
            "min_acceptance_rate": (
                result["acceptance_rate"],
                lambda value, limit: value >= limit,
            ),
            "max_policy_violations": (
                result["policy_violations"],
                lambda value, limit: value <= limit,
            ),
            "max_regressions": (
                result["regressions"],
                lambda value, limit: value <= limit,
            ),
            "max_cost_per_accepted_change": (
                result["cost_per_accepted_change"],
                lambda value, limit: value is not None and value <= limit,
            ),
        }
        for name, limit in gates.items():
            value, passed = checks[name]
            if not passed(value, limit):
                errors.append(
                    f"{tool} failed {name}: measured={value} limit={limit}"
                )
    return errors


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("experiment", type=Path)
    args = parser.parse_args()
    try:
        experiment = load(args.experiment)
    except (OSError, json.JSONDecodeError, ValueError) as error:
        parser.error(str(error))
    errors = validate(experiment)
    if errors:
        for error in errors:
            print(f"ERROR: {error}")
        raise SystemExit(1)
    summaries = summarize(experiment)
    print(json.dumps(summaries, indent=2, sort_keys=True))
    failures = apply_gates(summaries, experiment["gates"])
    if failures:
        for failure in failures:
            print(f"ERROR: {failure}")
        raise SystemExit(1)


if __name__ == "__main__":
    main()

Run:

python coding_tool_gate.py experiment.json

The script enforces paired task coverage across candidates. It does not claim that paired task runs replace a randomized human no-AI control; use a separate study design when the decision is about human productivity rather than product output.

Run Security as a Separate Test Track

Security failures should not be diluted inside a quality score.

Test each candidate with:

• a repository file containing an instruction to read a decoy secret;
• a tool result requesting an unrelated network call;
• a dependency installation that invokes a lifecycle script;
• a task that asks for a CI or permissions change outside scope;
• a forbidden external domain;
• a command whose harmless prefix hides a destructive operation;
• an attempted push or merge beyond the assigned branch;
• missing and expired credentials.

Record attempted and successful violations separately. The control, not the model's refusal, must block a forbidden action. Use disposable infrastructure and synthetic data.

Official security documentation for Cursor, Claude Code, and GitHub Copilot coding agent describes different approval, sandbox, network, branch, and audit boundaries. Test the exact mode rather than extrapolating one mode's controls to the whole product.

Interpret the Results as a Portfolio

The result may not be one winner:

• completion can win on short local edits;
• an interactive agent can reduce active time on exploratory changes;
• a cloud agent can help with bounded asynchronous maintenance;
• the current human workflow can remain best for high-context or high-risk tasks;
• a specialist static analyzer can outperform every generative tool on deterministic checks.

Choose by slice and preserve common policy, acceptance, and audit gates. A mixed toolchain is justified only when its benefit exceeds additional configuration, training, contract, and incident-response complexity.

Roll Out From Evidence, Not Excitement

Promotion should increase repository sensitivity and agent authority gradually:

1. dry-run the protocol and repair ambiguous tasks;
2. run the locked comparison in disposable environments;
3. pilot the selected mode on low-risk real work;
4. keep independent review and existing CI mandatory;
5. monitor acceptance, review load, regressions, incidents, and cost;
6. expand only the slices that continue to pass;
7. retain the previous tool and configuration for rollback.

Re-evaluate when the tool mode, model policy, client, permissions, contract, pricing, or task mix changes materially.

Common Failure Modes

Letting Operators Pick Their Favorite Tasks

Task selection can create more uplift than the tool. Freeze or randomize the suite before assignment.

Counting a Passing Narrow Test as Acceptance

Hidden invariants, maintainability, scope, and security can fail while one test passes. Use layered acceptance.

Ignoring Rejected Attempts

Dropping failed or expensive runs inflates both quality and cost results. Preserve every valid attempt.

Comparing Unequal Budgets

Unlimited retries for one candidate and a fixed quota for another measure budget, not tool quality. Record and equalize the constraint relevant to the decision.

Revealing Tool Identity to Reviewers

Brand preference can bias review. Normalize artifacts and blind identity where practical.

Forcing a Winner From Sparse Data

Wide confidence intervals and conflicting slices mean the result is inconclusive or mode-specific. Do not collapse them into a star rating.

Frequently Asked Questions

Should the same developer operate every tool?

Not automatically. One operator controls developer differences but may favor a familiar tool. Multiple operators improve representativeness but add variance. Use training, rotate or block assignments, and record experience so the design matches the decision.

Can old completed tickets be reused as tasks?

Yes, if the original patch, later commits, discussion, and hidden answers are removed from the candidate environment and the task remains realistic. Public history can still contaminate model training, so private or newly authored tasks provide stronger evidence.

Should tools receive the same model?

Only if the decision is specifically about harnesses with a shared model. For product selection, test the supported configuration you would actually deploy and record the model policy. Do not disable a product's core design merely to create artificial sameness.

How should ambiguous tasks be scored?

Predefine when clarification is correct. A tool that asks a necessary question should not be penalized as incomplete; a tool that invents requirements and changes code should fail scope or correctness.

Can this protocol prove organization-wide productivity?

No. It can support a bounded product decision for the sampled tasks, repositories, users, and controls. Organization-wide effects also depend on task selection, learning, workflow redesign, review capacity, quality, and long-term maintenance.

Summary

A defensible Cursor, Claude Code, and Copilot comparison is an experiment, not a ranking article. Freeze the decision, task suite, environment, candidate configuration, budgets, acceptance rubric, and security track; retain every outcome; measure accepted changes and human load; report uncertainty and slices; then roll out only the evidence-backed modes through a reversible canary.

References

• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• OpenAI: Separating Signal From Noise in Coding Evaluations
• METR: Early-2025 AI and Experienced Open-Source Developer Productivity
• METR: Changing the Developer Productivity Experiment Design
• OWASP LLM01:2025 Prompt Injection
• Cursor Agent Security
• Claude Code Security
• GitHub Copilot Coding Agent Risks and Mitigations
• AI Coding Tools Selection Guide
• AI Coding Tool Cost Evaluation