AI Agent Observability: Privacy-Safe Traces, Evaluation, and Cost

The Design Goal

Agent observability should answer three different questions:

1. What happened? Protocol, state, policy, tool, latency, and error evidence.
2. Did it achieve the task? Deterministic checks, scenario evaluation, human review, and calibrated judges.
3. What did it cost? Reconciled provider usage, downstream charges, retries, and budget outcomes.

These questions need different data and retention. Putting all raw model input and output into one permanent trace increases privacy and security risk without guaranteeing better debugging.

An Event Contract, Not a Transcript

Define an application-owned event schema. A bounded event can look like:

{
  "event": "tool_call_completed",
  "schema_version": 3,
  "run_id": "run_7f...",
  "request_id": "req_21...",
  "trace_id": "4bf92f...",
  "step": 4,
  "tenant_hash": "h:...",
  "subject_hash": "h:...",
  "tool": "invoice.list",
  "tool_version": "2026-06-1",
  "argument_digest": "sha256:...",
  "policy": "allow",
  "state_version": 12,
  "duration_ms": 184,
  "result_class": "bounded_page",
  "error_code": null,
  "usage": {
    "input_tokens": 812,
    "output_tokens": 164,
    "estimated_cost_usd": 0.0019,
    "price_table_version": "2026-06-01"
  }
}

The event is evidence, not a claim that the model’s private reasoning was captured. Keep trace_id, run_id, and request_id distinct: one trace can contain many runs, and a request can be retried.

Recommended Fields

Never put access tokens, authorization headers, raw secrets, hidden reasoning, or unrestricted user content into a default span.

Redaction and Retention

Privacy is part of the observability design:

• classify fields before instrumentation;
• redact at the producer, not only in a downstream dashboard;
• hash identifiers with a tenant-appropriate key and document rotation;
• bound strings, arrays, tool results, and exception messages;
• keep raw fixtures in a separate access-controlled store when a replay is justified;
• define retention by purpose and legal basis;
• propagate deletion to traces, indexes, caches, backups, exports, and evaluation datasets;
• test that denied requests do not leak object existence through telemetry.

Sampling is not a privacy policy. A 1% sample can still contain a sensitive record, while a security event may require complete metadata without raw content.

Trace Semantics

Use spans or events for observable transitions:

run_started
  -> model_call_completed
  -> tool_call_requested
  -> policy_decision
  -> tool_call_completed
  -> state_updated
  -> run_finished

Record the policy decision and outcome separately. A model-proposed Tool call is not an executed Tool call. A successful HTTP response is not proof that the business action was authorized or correct.

Loop detection can use bounded evidence:

from dataclasses import dataclass
from typing import Iterable

@dataclass(frozen=True)
class StepEvent:
    step: int
    tool: str | None
    argument_digest: str | None
    state_version: int
    policy: str
    outcome: str

def repeats(events: Iterable[StepEvent], window: int = 4) -> bool:
    items = list(events)
    if window <= 0 or len(items) < window * 2:
        return False
    signature = lambda event: (
        event.tool,
        event.argument_digest,
        event.state_version,
        event.policy,
        event.outcome,
    )
    return [signature(e) for e in items[-window * 2:-window]] == [
        signature(e) for e in items[-window:]
    ]

This identifies repeated observable behavior. It does not infer hidden model reasoning or prove that a loop is malicious.

Evaluation Is Measurement

Use several evaluation layers with explicit oracles:

Do not collapse these into a single “quality score.” A high judge score cannot override a failed authorization invariant.

Calibrating an LLM Judge

If a judge is useful for triage:

1. define a rubric and allowed evidence;
2. create positive, negative, ambiguous, and adversarial examples;
3. compare it with independent human labels;
4. measure agreement by task slice, not only a global average;
5. test prompt, model, and rubric changes as evaluator regressions;
6. keep the judge’s result separate from the authoritative pass/fail oracle.

Do not ask a judge to reconstruct hidden reasoning. Evaluate observable claims, citations, tool outcomes, policy events, and task results.

Cost Accounting

A cost ledger should preserve source and uncertainty:

{
  "run_id": "run_7f...",
  "currency": "USD",
  "input_tokens": 812,
  "output_tokens": 164,
  "cached_tokens": 0,
  "provider_charge": 0.0017,
  "tool_charge": 0.0002,
  "retry_charge": 0,
  "estimated": false,
  "price_table_version": "provider-2026-06-01"
}

Reconcile by run, tenant, model, task type, tool, and release. Separate user-visible billing from internal estimates. Account for retries, canceled work, cache hits, streaming, currency conversion, and downstream services.

Cost controls should be budgets and policy:

• per-request and per-run token/step/time limits;
• per-principal and tenant quotas;
• model routing tested against quality and latency;
• cache only when privacy, freshness, and key scope are correct;
• approval for expensive or external side effects;
• alerts based on workload baselines and budget commitments.

There is no universal saving percentage. Measure the workload before and after a change.

Tool and Framework Choice

Langfuse, LangSmith, Phoenix, Helicone, Braintrust, and OpenTelemetry differ in hosting, data model, integrations, retention, evaluation features, and pricing. A useful selection record includes:

Use the least data and smallest integration that answers the current operational question. Re-evaluate vendors as versions and pricing change.

A Risk-Ordered Rollout

Stage 1: Contract and privacy

Define event schemas, redaction, retention, deletion, access roles, and budget fields before adding dashboards.

Stage 2: Runtime evidence

Instrument initialization, model calls, Tool proposals, policy decisions, execution outcomes, state changes, cancellations, and errors with bounded fields.

Stage 3: Evaluation

Create a small redacted scenario set, deterministic oracles, abuse cases, replay comparisons, and a human review path. Add a judge only after its disagreement is measurable.

Stage 4: Cost and operations

Reconcile provider invoices, set workload-specific budgets, monitor tail latency and downstream saturation, and test exporter failure without blocking the agent unexpectedly.

Stage 5: Release gates

Require contract, scenario, replay, abuse, privacy, and cost evidence for changes to prompts, models, tools, policies, and instrumentation.

What Observability Cannot Prove

• A trace cannot prove that a model’s reasoning was truthful.
• A successful Tool span cannot prove object authorization.
• A judge score cannot prove safety for unseen inputs.
• A token count cannot prove the provider invoice.
• A dashboard cannot replace deletion, access control, or incident response.
• An OpenTelemetry exporter cannot define application semantics for you.

Production Checklist

• [ ] Own and version the event contract.
• [ ] Redact and bound at the producer.
• [ ] Keep hidden reasoning, tokens, secrets, and raw user content out of default spans.
• [ ] Separate trace, run, and request identifiers.
• [ ] Record policy decisions and business outcomes independently.
• [ ] Use contract, scenario, replay, abuse, human, and calibrated judge evaluation.
• [ ] Keep authoritative oracles separate from model scores.
• [ ] Reconcile usage with a versioned price table and uncertainty flag.
• [ ] Propagate deletion through telemetry and evaluation stores.
• [ ] Test exporter outages, retention, access, rollback, and budget enforcement.

Conclusion

Good Agent observability is not a transcript of everything the model saw or “thought.” It is a privacy-aware evidence system: bounded events explain execution, layered evaluation measures outcomes, and a reconciled ledger controls cost. Design those contracts before choosing a dashboard, then keep every claim proportional to the evidence it can actually provide.

Primary Sources

• OpenTelemetry Documentation
• Model Context Protocol Specification
• NIST AI Risk Management Framework
• W3C Trace Context