Agent Reliability: pass@k, pass^k, and Failure Traces

TL;DR

Repeat each task from a clean starting state and report average success, at-least-one success, and all-runs success separately. Estimate metrics per task before aggregation; do not raise the overall average to a power. Analyze uncertainty and behavior traces, and remember that consistent output can still be consistently wrong.

One successful demonstration is not reliable delivery

Repeated reliability asks whether the same task continues to succeed from the same starting conditions. One demonstration establishes the existence of a successful trajectory, not the probability of reproducing it.

The September 2026 Consistency Gap study uses 168 AppWorld tasks with five runs per task in a specified setup. It reports approximately 77% mean performance for GPT-4.1 and approximately 53% success across all five runs, a gap of 24.4 percentage points. These results belong to that agent, model, dataset, and configuration. They are not a production success estimate for all agents.

The gap does not mean the model became worse between measurements. The metrics ask different questions. Mean success credits every successful run; all-runs success exposes a failure within a task's repetitions. For automated order processing, an intermittent authorization failure can matter more than a small aggregate score change.

An agent harness determines tools, retries, context, and stopping. Repeated trials evaluate the combined system. Before attributing variation to sampling, establish that the environment, contract, and grader remain stable.

Three metrics answer three operational questions

Mean success measures the success fraction across attempts. pass@k measures at least one success among k attempts. pass^k measures success on all k attempts. A trial is a complete run from a specified initial state to termination; independent acceptance determines success.

The 2024 τ-bench paper introduced pass^k for reliability in tool-and-user interaction tasks. Acceptance involves final database state and relevant task requirements. The method predates recent reliability studies; pass^k is not a new 2026 invention.

For task i with fixed success probability p_i, assuming independent trials within that task:

Mean = average(p_i)
pass@k = average(1 - (1 - p_i)^k)
pass^k = average(p_i^k)

The task-level ordering matters. If two tasks have success probabilities 1 and 0, mean success is 0.5, and average pass^2 remains 0.5. Squaring the overall mean gives 0.25 and erases task heterogeneity. Even the per-task formulas require assumptions that repeated execution must examine.

Estimate from finite observations

Given n observed trials and c successes for each task, combination ratios estimate at-least-one and all-success events when selecting k observed trials without replacement. Require 1 ≤ k ≤ n and a consistent success definition.

pass@k estimate = 1 - C(n-c, k) / C(n, k)
pass^k estimate = C(c, k) / C(n, k)

A combination is zero when fewer than k items are available. Under independent, identically distributed trials, these estimators can estimate the corresponding repeated-event probabilities. They do not repair dependence or environmental contamination.

The following data are synthetic; each task has five runs:

A high pass@2 does not establish stable execution. It also presumes that a successful candidate can be recognized. If the deployed selector sometimes chooses the wrong candidate, include selection in end-to-end evaluation rather than reporting an oracle's best result.

When tasks have unequal repetition counts, calculate each task's metric first and then average with declared weights. Pooling all trials gives more weight to tasks that happened to run more often. If business traffic should determine weighting, state the traffic distribution explicitly and report a task-weighted view as well.

Runnable Go: calculate ratios without huge combinations

This Go 1.23 example uses products of ratios instead of directly calculating potentially enormous binomial coefficients. It requires only the standard library and rejects invalid counts.

package main

import "fmt"

type Metrics struct {
	Mean    float64
	PassAny float64
	PassAll float64
}

func subsetRatio(selected, total, k int) float64 {
	if selected < k {
		return 0
	}
	ratio := 1.0
	for j := 0; j < k; j++ {
		ratio *= float64(selected-j) / float64(total-j)
	}
	return ratio
}

func estimate(n, c, k int) (Metrics, error) {
	if n < 1 || c < 0 || c > n || k < 1 || k > n {
		return Metrics{}, fmt.Errorf("require n>=1, 0<=c<=n, 1<=k<=n")
	}
	return Metrics{
		Mean:    float64(c) / float64(n),
		PassAny: 1 - subsetRatio(n-c, n, k),
		PassAll: subsetRatio(c, n, k),
	}, nil
}

func main() {
	counts := []int{5, 4, 2, 0}
	sum := Metrics{}
	for i, c := range counts {
		m, err := estimate(5, c, 2)
		if err != nil {
			panic(err)
		}
		fmt.Printf("%c mean=%.3f any=%.3f all=%.3f\n",
			'A'+i, m.Mean, m.PassAny, m.PassAll)
		sum.Mean += m.Mean
		sum.PassAny += m.PassAny
		sum.PassAll += m.PassAll
	}
	size := float64(len(counts))
	fmt.Printf("macro mean=%.3f any=%.3f all=%.3f\n",
		sum.Mean/size, sum.PassAny/size, sum.PassAll/size)
	if _, err := estimate(5, 6, 2); err == nil {
		panic("invalid count accepted")
	}
}

Run go run main.go:

A mean=1.000 any=1.000 all=1.000
B mean=0.800 any=1.000 all=0.600
C mean=0.400 any=0.700 all=0.100
D mean=0.000 any=0.000 all=0.000
macro mean=0.550 any=0.675 all=0.425

The function computes metrics, not confidence intervals or trial independence. Very small probabilities can underflow; ordinary finite repetition counts are suitable for this implementation, while extreme tail calculations need log-space arithmetic and a precision statement.

Five observed successes do not prove an underlying success probability of one. A printed 1.000 is an estimate from finite data, not a deterministic guarantee. Choose repetition counts from acceptable failure risk and estimation precision, rather than selecting a k that produces an attractive score.

Reset the starting environment for every trial

Trials need isolated files, databases, caches, and external-operation identities to distinguish decision variation from state contamination. Failed trials require cleanup too; partial completion can make later runs artificially easier.

Freeze task definitions without pretending that all randomness disappears. Record model versions, sampling parameters, tool versions, and controllable seeds. Temperature zero does not guarantee identical behavior across infrastructure changes, model updates, or parallel tool scheduling.

Timeouts and tool outages count against system success, but should also appear in root-cause slices. A separate experiment with frozen tool responses can isolate decision behavior. It must not be presented as online system reliability. The long-running recovery article covers interruptions across session boundaries.

For uncertainty, a task-level bootstrap resamples tasks with replacement, carries each sampled task's full trial set, recalculates macro metrics, and reports a preselected percentile interval. When comparing systems, resample the same tasks together. This primarily captures task-sampling uncertainty; within-task stochasticity requires sufficient repetitions and, where appropriate, a hierarchical analysis.

Diagnose instability through behavior traces

Failure analysis should align actions and state transitions, not merely compare final wording. An agent trajectory records tool choices, arguments, observations, and termination reasons.

For a task with both successful and failed trials, align semantic stages: parse dates, query orders, check authorization, and present results. Then inspect the earliest consequential divergence. One trial may omit a permission query, another may parse a date differently, and another may stop after a recoverable tool error.

The earliest divergence is a diagnostic lead, not automatic causal proof. Different tool names can produce equivalent correct behavior, and identical calls can read different external state. Inspect arguments and observations, then run an intervention such as freezing the date parser to see whether the failure rate changes.

Retest fixes on the same task set with independent acceptance. Shorter traces or more uniform explanations do not necessarily improve reliability. Text similarity should not substitute for business correctness.

FAQ

Is high pass@k enough for deployment?

Only if the deployed workflow actually makes k attempts, has a reliable selector, and meets the cost budget. A single-attempt product cannot inherit the reported benefit of oracle candidate search.

How should I choose k?

Match the repeated-use scenario. Reporting k=1, 2, and 5 can expose the curve, provided n reaches the largest k. High-risk systems also need severe-error rates; aggregate pass^k can hide rare authorization failures.

Can identical outputs still be unreliable?

Yes. Repeatability measures variation; acceptance measures correctness. An agent can produce the same wrong outcome on every run. Report both rather than treating them as interchangeable.

How do I compare two harnesses?

Use matched tasks, snapshots, and graders, then compare task-level metrics and budgets. Controlled harness evaluation explains how to avoid attributing extra attempts to architectural improvement.

References

• Consistency Gap study, September 8, 2026, repeated runs in a specified AppWorld configuration.
• τ-bench paper, 2024, tool-and-user evaluation and pass^k.
• Anthropic: Demystifying evals for AI agents, January 9, 2026.