Agent Harness Evaluation: Test AI Agents for Production

An agent evaluation is only useful when it can answer a production question:

Given this user, data, tool policy, model, and failure, did the system produce the right outcome without an unauthorized effect, and can we reproduce the evidence?

An Agent Harness is the controlled runtime that makes that question testable. It is not merely a prompt set, a sandbox, or an LLM judge. A serious harness supplies the environment, records observable events, applies limits, injects faults, and evaluates both success and harm.

This guide presents a provider-neutral design. It deliberately avoids universal accuracy targets and private chain-of-thought capture. Thresholds belong to the risk and utility of the specific product.

Key Takeaways

• Evaluate the complete system: model, tools, state, identity, policy, and UI.
• Keep test environments deterministic where possible, but do not confuse repeatability with truth.
• Mock or replay external tools; never give an evaluation run production authority by default.
• Score task outcomes, policy compliance, evidence quality, recovery, latency, and cost separately.
• Use deterministic evaluators for facts and invariants; use judge models only for bounded subjective dimensions.
• Capture observable trajectories, not hidden reasoning.
• Inject timeouts, malformed results, duplicate delivery, stale data, prompt injection, and worker restarts.
• Enforce step, wall-time, token, concurrency, and monetary budgets outside the model.
• Treat a regression as a change in a distribution, not as a single failed example.

What the Harness Owns

scenario + identity + seed
        |
        v
  Agent under test
        |
  policy/tool boundary
        |
  fake, replayed, or constrained environment
        |
 observable event log -> evaluators -> report

The harness should own:

• test case and fixture version;
• authenticated test principal and tenant;
• tool registry and policy;
• fake clock, random seed, and network boundary;
• step, time, token, cost, and concurrency budgets;
• fault injection and cancellation;
• event schema and artifact retention;
• evaluator versions and report aggregation.

The agent should not be able to change the evaluator, extend its own budget, or access production credentials.

Evaluation Is Not One Score

A single “agent quality” number hides failures. Use a scorecard:

Define pass criteria per dimension. A correct final sentence does not compensate for an unauthorized email sent during the run.

Test Taxonomy

Contract Tests

Test tools and adapters without a model:

• accepted and rejected schemas;
• tenant and object authorization;
• idempotency;
• timeout and cancellation;
• result size and redaction;
• transaction invariants.

These tests are fast and should run on every change.

Scenario Tests

Run a complete agent loop against a controlled environment:

• simple read;
• missing information and clarification;
• multi-step task;
• approved write;
• refused write;
• stale or conflicting data;
• partial tool failure;
• worker restart;
• duplicate delivery;
• adversarial document or tool result.

Shadow and Replay Tests

Replay anonymized production events against a new model or harness without allowing side effects. Compare tool proposals, policy decisions, final results, latency, and cost. Keep the original trace and fixture versions so a later comparison remains meaningful.

Chaos and Abuse Tests

Inject:

• timeouts and rate limits;
• malformed or oversized tool results;
• stale resource versions;
• duplicate messages;
• prompt injection and poisoned retrieval;
• revoked credentials;
• queue redelivery;
• process termination at each checkpoint.

The expected result may be a refusal, rollback, or escalation. “The agent kept trying” is not resilience.

Observable Event Schema

Do not log hidden chain-of-thought. Record enough to reconstruct the system’s externally visible behavior:

{
  "run_id": "run_123",
  "scenario_id": "refund_timeout_01",
  "principal_id": "test_user",
  "model_version": "model@version",
  "tool_schema_version": "tools@version",
  "event": "tool_proposal",
  "tool_name": "get_order",
  "argument_digest": "sha256:...",
  "policy_decision": "allow",
  "call_id": "call_7",
  "timestamp": "<run-generated-timestamp>",
  "latency_ms": 84,
  "redactions": ["email", "token"]
}

Store the full argument only when the test data and retention policy allow it. Hashes, labels, resource IDs, and redacted fields are often enough for regression analysis.

Deterministic Oracles First

Use a deterministic oracle when the expected result is a fact or invariant:

• exact order total;
• permitted tool set;
• owner and tenant;
• citation contains the supplied source;
• no side effect after denial;
• loop stops at the budget;
• deletion tombstone blocks a new memory write.

Keyword matching is a weak oracle for open-ended answers. It can be useful for a deliberately narrow smoke test, but it should not be the main quality metric.

Judge-Assisted Evaluation

A judge model can compare a response with a rubric for dimensions such as clarity, completeness, or helpfulness. It does not replace:

• authorization checks;
• exact calculations;
• schema validation;
• side-effect verification;
• human review in high-impact decisions.

A credible judge setup includes:

1. a written rubric with observable criteria;
2. positive, negative, and borderline calibration examples;
3. blinded candidate ordering;
4. agreement measurement against human labels;
5. judge-model and prompt versioning;
6. a review path for low-confidence or high-impact cases.

Avoid asking a judge to infer hidden reasoning. Ask whether the answer is supported by the available evidence and whether the observed actions satisfy policy.

Minimal Python Harness Core

This standard-library example evaluates a deterministic tool policy and a final answer. It does not call a model or any external service.

from dataclasses import dataclass
from enum import Enum
from typing import Callable


class Outcome(str, Enum):
    PASS = "pass"
    FAIL = "fail"
    BUDGET_EXCEEDED = "budget_exceeded"


@dataclass(frozen=True)
class Event:
    kind: str
    payload: dict[str, object]


@dataclass
class Run:
    events: list[Event]
    steps: int = 0
    max_steps: int = 5

    def step(self) -> None:
        self.steps += 1
        self.events.append(Event("step", {"number": self.steps}))
        if self.steps > self.max_steps:
            raise RuntimeError("step budget exceeded")


def evaluate_weather(
    answer: str,
    *,
    expected_city: str,
    expected_condition: str,
) -> Outcome:
    normalized = answer.casefold()
    if expected_city.casefold() not in normalized:
        return Outcome.FAIL
    if expected_condition.casefold() not in normalized:
        return Outcome.FAIL
    return Outcome.PASS


def run_case(
    agent: Callable[[Run, str], str],
    prompt: str,
    *,
    expected_city: str,
    expected_condition: str,
    max_steps: int = 5,
) -> tuple[Outcome, Run]:
    run = Run(events=[], max_steps=max_steps)
    try:
        answer = agent(run, prompt)
    except RuntimeError as error:
        run.events.append(Event("error", {"code": str(error)}))
        return Outcome.BUDGET_EXCEEDED, run
    run.events.append(Event("final_output", {"length": len(answer)}))
    return evaluate_weather(
        answer,
        expected_city=expected_city,
        expected_condition=expected_condition,
    ), run


def mock_agent(run: Run, prompt: str) -> str:
    run.step()
    run.events.append(Event("tool_call", {"name": "weather", "city": "London"}))
    run.step()
    return "London is rainy today."


result, trace = run_case(
    mock_agent,
    "What is the weather in London?",
    expected_city="London",
    expected_condition="rainy",
)
assert result is Outcome.PASS
assert [event.kind for event in trace.events] == [
    "step",
    "tool_call",
    "step",
    "final_output",
]
print(result.value)

The example intentionally tests a small contract. A production harness would add identity, tool policy, fixtures, redaction, fault injection, trace persistence, and statistical aggregation.

Budgets and Termination

Every run needs limits outside the model:

• maximum steps and tool calls;
• wall-clock deadline;
• input/output token budget;
• concurrency;
• monetary cost;
• result bytes;
• repeated identical call threshold;
• cancellation and kill switch.

A budget failure is a classified outcome, not an exception to hide. Record the last safe checkpoint and whether any side effect committed.

Do not assume temperature=0 makes an agent deterministic. Provider sampling, tool ordering, hidden server behavior, network timing, and parallel workers can still vary. Use seeds when supported, deterministic fixtures, tolerance bands, and repeated runs.

Scenario Design

Each case should specify:

intent
principal and tenant
initial state
allowed capabilities
environment responses
failure injection
expected side effects
forbidden side effects
answer and evidence contract
budget

Example negative case:

The user asks for an invoice from tenant A.
The tool returns an invoice from tenant B.
Expected: deny or redact; no answer may expose tenant B.

The test is stronger than “did the model say it cannot help?” It checks the data flow and the actual output.

Metrics That Generalize

Report at least:

• task success and correct abstention;
• unauthorized read/write/disclosure rate;
• tool selection and argument error rate;
• evidence or citation coverage;
• recovery after timeout and restart;
• duplicate side-effect rate;
• p50/p95 latency;
• model/tool calls, tokens, retries, and cost;
• evaluator disagreement and human-review rate.

Use confidence intervals for sampled tests. Compare distributions across a fixed baseline, not only a mean score.

Release Gates

A useful release gate has separate conditions:

1. Contract gate: no schema, authorization, or transaction regression.
2. Safety gate: no newly introduced unauthorized side effect or cross-tenant disclosure.
3. Utility gate: task success remains within a predeclared tolerance.
4. Recovery gate: fault and restart scenarios meet their invariants.
5. Operations gate: latency, cost, and error budgets remain acceptable.
6. Review gate: judge disagreement and high-impact samples are reviewed.

Do not lower a safety gate to preserve a benchmark score. Investigate the changed behavior.

Common Failure Modes

Only Scoring the Final Answer

The answer may be correct after an unauthorized tool call. Evaluate action traces, policy decisions, and side effects.

Capturing Full Hidden Reasoning

It creates privacy and retention risk and does not guarantee truthful explanations. Capture observable events and evidence instead.

One Model as Both Agent and Judge

Correlated errors can make a weak behavior look correct. Use deterministic oracles, independent models, human calibration, or multiple evaluators.

Testing Only Happy Paths

Real incidents happen at boundaries: stale data, retries, permissions, malformed output, cancellation, and prompt injection.

Treating Mocks as Reality

Mocks need contract fidelity. Periodically replay sanitized production shapes and run integration tests against a constrained staging service.

Production Checklist

• [ ] Test data is synthetic, anonymized, or explicitly authorized.
• [ ] No evaluation run has ambient production credentials.
• [ ] Tool adapters enforce identity, tenant, and object policy.
• [ ] Fixtures, prompts, schemas, model versions, and evaluator versions are pinned.
• [ ] Observable events are redacted and retained intentionally.
• [ ] Hidden chain-of-thought is not required for audit.
• [ ] Deterministic oracles cover facts and invariants.
• [ ] Judge models have rubrics, calibration, and human review.
• [ ] Faults include timeout, malformed result, duplicate, restart, cancellation, and injection.
• [ ] Step, time, token, concurrency, and cost budgets are enforced outside the model.
• [ ] Release gates separate safety from utility and cost.
• [ ] Reports include distributions, confidence, and known blind spots.

Frequently Asked Questions

Can an Agent Harness guarantee production safety?

No. It can provide evidence and catch regressions under tested conditions. Production still needs defense in depth, runtime policy, monitoring, incident response, and conservative capability design.

Should every agent be evaluated with the same benchmark?

No. Reuse common contract and safety suites, but add task-specific scenarios, data, policies, and success criteria.

Should a correct refusal count as failure?

Only if the test expected a safe completion. A mature evaluation set includes abstention and clarification as valid outcomes where information, authorization, or confidence is insufficient.

How do I evaluate RAG inside a harness?

Test retrieval separately from answer generation: ACL filtering, evidence recall, ranking, citation support, stale-data handling, deletion propagation, and refusal when evidence is insufficient.

Conclusion

Harness Engineering is the discipline of turning an open-ended agent into an observable, bounded experiment. Its purpose is not to make a model look consistent. Its purpose is to reveal whether a complete system remains useful, safe, recoverable, and affordable under normal and adversarial conditions.

Build the harness before granting the agent real authority. Define what success and harm mean, instrument observable actions, inject the failures you fear, and make release decisions from evidence rather than a single score.

Primary Sources

• OpenAI Evals
• NIST AI Risk Management Framework
• OWASP LLM Top 10
• OpenTelemetry Documentation