Evaluator-Driven Agent Harnesses: Feedback and Stopping

TL;DR

An evaluator-driven harness separates task contracts, candidate versions, and acceptance evidence. Feedback should identify reproducible defects. Revision needs a budget, stalled-progress detection, and retention of the best valid candidate. Deterministic checks take precedence over model scores, while held-out acceptance detects grader overfitting. The loop does not guarantee convergence.

Adding a judge does not create a reliable loop

A separate evaluator helps separate generation from acceptance, but another model call does not establish trustworthy feedback. A generator can submit an application that does not run, receive a favorable score based on a screenshot, and then break login while improving its colors. A controller that tracks only the score mistakes regression for progress.

An agent harness must control candidate identity, acceptance evidence, spending, and stopping reasons. An LLM-as-a-Judge can evaluate qualities that are difficult to encode completely. It should not replace direct checks of functional requirements.

Anthropic's 2026 application-development experiment separates a generator and evaluator, uses examples to calibrate design judgments, and later organizes planning, generation, and evaluation around sprint contracts. Its dimensions include design quality, originality, craft, and functionality. The account also describes diminishing feedback returns and intermediate results that can be better than the final result. These observations reflect particular models and author preferences, not a universal aesthetic rubric or a convergence theorem.

For an order-search application, correct filtering, clear error states, and usable presentation are separate requirements. Compressing them into “quality: 8” removes the information needed to repair the application and justify acceptance.

Define the contract before the score

A task contract specifies required behavior, acceptable evidence, and acceptance authority. A score is one component of that contract; it must not silently redefine completion during execution.

Hard requirements are gates: compilation, access control, and data correctness. Soft dimensions use anchored ratings, such as whether an error message helps a user recover. A high visual score must not compensate for a failed authorization check. Examples should anchor rating levels so that “usable but rough” does not overlap with “cannot be operated.”

Anthropic's evaluation guide distinguishes tasks, trials, graders, and actual outcomes. Here the distinction between a favorable judgment and an application that really works is essential. A grader that cannot inspect execution can certify only what it observed.

Changing the contract creates a new version. Adding mobile acceptance midway through a run is a requirements change, not a score correction. Recheck candidates under the revised contract. Conflicting requirements should produce escalation rather than repeated generation against an impossible target.

Feedback should identify reproducible defects

Useful feedback connects expected behavior, observed behavior, evidence, and the candidate being evaluated. A total score alone is rarely a repair instruction.

{
  "candidateId": "candidate-17",
  "contractVersion": "order-app-v3",
  "hardChecksPassed": false,
  "findings": [
    {
      "checkId": "date-filter",
      "expected": "exclude orders outside selected range",
      "observed": "one out-of-range order remains",
      "evidenceId": "browser-run-118",
      "reproduction": "fixture B, select 2026-10-01 to 2026-10-07"
    }
  ],
  "suggestedAction": "revise"
}

Development feedback can include detailed reproduction conditions. Final held-out acceptance should not continuously feed hidden answers into revision. Once a team repeatedly inspects held-out failures and repairs against them, those tasks become development data and need replacement.

Evaluator failures require their own status. Missing evidence, an unavailable environment, and a malformed judge response are not equivalent to poor candidate quality. Bind each report to an immutable candidate so that a delayed result cannot accidentally guide repairs to a different version.

An evaluator may propose a repair direction, but the observation remains the durable part of the report. “Exclude this order” is grounded in evidence; “rewrite the persistence layer” is a hypothesis. Labeling that distinction prevents an uncertain diagnosis from becoming an unnecessary large change.

Give the loop explicit exit paths

A controller needs acceptance, revision, escalation, and stopping outcomes. It should preserve the best valid candidate rather than assuming the last candidate is the best.

A plateau means several evaluations fail to meet a predefined minimum-progress rule. It limits cost; it does not prove that the task is impossible. Before a candidate passes hard checks, progress may mean removing specific blockers. After validity is established, soft-quality improvements become comparable. Fixing many cosmetic issues must not offset one unresolved data leak.

The budget includes the generator, evaluator, tools, retries, and browser execution. Reserve budget before dispatch and reconcile actual usage afterward. Parallel calls need shared reservations. Counting only generator tokens understates feedback-loop cost and can create uncontrolled spending.

Keep the progress reference separate from the best candidate. Several small improvements can accumulate into meaningful progress, even when no individual step exceeds the minimum gain. Candidate selection and plateau detection answer different questions.

Runnable Go: budget, plateau, and candidate retention

This Go 1.23 controller processes a synthetic sequence of completed evaluations. Costs are illustrative ledger units, not model prices, and scores are demonstration inputs rather than benchmark results.

package main

import (
	"fmt"
	"math"
)

type Result struct {
	ID       string
	Valid    bool
	Score    float64
	Cost     int
	Conflict bool
}

type Controller struct {
	Budget      int
	Spent       int
	Patience    int
	Stale       int
	MinGain     float64
	Target      float64
	Best        *Result
	ProgressRef float64
}

func (c *Controller) observe(r Result) (string, error) {
	if r.ID == "" || r.Cost <= 0 || math.IsNaN(r.Score) ||
		math.IsInf(r.Score, 0) || r.Score < 0 || r.Score > 1 {
		return "", fmt.Errorf("invalid evaluation result")
	}
	if c.Spent+r.Cost > c.Budget {
		return "stop:budget", nil
	}
	c.Spent += r.Cost
	if r.Conflict {
		return "escalate:contract", nil
	}
	progress := false
	if r.Valid {
		if c.Best == nil || r.Score > c.Best.Score {
			copy := r
			c.Best = &copy
		}
		if r.Score >= c.ProgressRef+c.MinGain {
			c.ProgressRef = r.Score
			progress = true
		}
		if r.Score >= c.Target {
			return "candidate:heldout", nil
		}
	}
	if progress {
		c.Stale = 0
	} else {
		c.Stale++
	}
	if c.Spent == c.Budget {
		return "stop:budget", nil
	}
	if c.Stale >= c.Patience {
		return "stop:plateau", nil
	}
	return "revise", nil
}

func main() {
	c := Controller{
		Budget: 12, Patience: 2, MinGain: 0.02, Target: 0.9,
	}
	results := []Result{
		{ID: "a", Valid: true, Score: 0.70, Cost: 3},
		{ID: "b", Valid: true, Score: 0.71, Cost: 3},
		{ID: "c", Valid: false, Score: 0.95, Cost: 3},
	}
	for _, r := range results {
		action, err := c.observe(r)
		if err != nil {
			panic(err)
		}
		fmt.Println(r.ID, action)
		if action != "revise" {
			break
		}
	}
	if c.Best != nil {
		fmt.Printf("retained=%s score=%.2f spent=%d\n",
			c.Best.ID, c.Best.Score, c.Spent)
	}
}

Run go run main.go:

a revise
b revise
c stop:plateau
retained=b score=0.71 spent=9

Candidate c scores higher but fails the hard gate, so it cannot displace b. Candidate b does not improve enough to reset patience, yet it is better than a and remains available. The progress reference allows small gains to accumulate.

This entrance processes results after evaluation. It does not prevent a real API call from exceeding budget. Production dispatch must reserve an upper bound first, then settle errors, cancellation, and timeouts. candidate:heldout requests final acceptance; it does not mean accepted. A retained checkpoint also must not be presented as a completed task.

Show that feedback improves independent outcomes

Feedback is useful when it improves acceptance outside the grader repeatedly consulted during revision. Separate the value of feedback from the value of additional inference spending.

Record development scores, hard-check pass rates, held-out acceptance, cumulative cost, and regressions at every round. If development scores rise while held-out quality falls, investigate overfitting, grader drift, and selection bias. Repeatedly choosing the maximum noisy score also requires independent confirmation.

Some examples in Anthropic's account compare a single agent with a much longer and more expensive full-harness run. They cannot establish an equal-cost improvement. The controlled evaluation and ablation article explains how to separate budget, model, and harness contributions.

Human review should sample both accepted and rejected candidates. Reviewing only accepted outputs misses false rejection; reviewing only failures misses false acceptance. Freeze the grader within an experiment, recalibrate after changes, and preserve the evidence used for each decision.

FAQ

Can the generator and evaluator use the same model?

Yes, but different roles do not guarantee independent mistakes. Combine role separation with deterministic checks, calibration examples, and stratified human review. Different system prompts are not independent facts.

Should a plateau discard the task?

No. Preserve the valid candidate, unmet requirements, and cost record. Replan, reduce scope, change tools, or escalate. A plateau diagnoses the current strategy, not every possible strategy.

Should the evaluator write the repair?

Usually it should first report defects and evidence. Writing a repair makes the evaluator part of generation and changes attribution. A separate repair stage is possible, but final acceptance should remain independent.

How does the loop survive a session boundary?

Persist contracts, candidates, and acceptance evidence, then verify the actual artifact on recovery. The long-running handoff article provides that protocol; saving only the last judge comment is insufficient.

References

• Anthropic: Harness design for long-running application development, March 24, 2026.
• Anthropic: Demystifying evals for AI agents, January 9, 2026.
• LLM-as-a-Judge calibration and limitations.