TL;DR

Production RAG evaluation must identify where a system failed. Measure retrieval, generation, end-to-end task outcome, safety, and operations separately. Use a versioned test set with realistic slices, calibrate LLM-as-judge metrics against human labels, and gate releases with both aggregate and critical-slice rules. A high average score must never hide missing evidence, unsupported claims, or cross-tenant retrieval.

Why one RAG score is not enough

A RAG system can fail before or after the model receives context. A single satisfaction score says that something went wrong but cannot tell the team what to change.

flowchart LR A["Question"] --> B["Query processing"] B --> C["Retrieval"] C --> D["Reranking and filtering"] D --> E["Generation"] E --> F["Answer and citations"] C --> G["Retrieval metrics"] E --> H["Generation metrics"] F --> I["Task and safety metrics"]
Layer Core question Example signals
Retrieval Did the system find and rank needed evidence? recall@k, precision@k, MRR, nDCG, context recall
Generation Did the answer use the supplied evidence correctly? answer faithfulness, answer relevance, citation entailment
End to end Did the user receive a correct and useful result? task success, reference correctness, abstention quality
Safety Did policy and data boundaries hold? unauthorized retrieval, sensitive leakage, unsafe action
Operations Is the quality affordable and reliable? p95 latency, cost, error rate, token and context budget

The RAGAS paper formalized reference-light evaluation across faithfulness, answer relevance, and context relevance. These are useful components, not a universal production scorecard.

Build a test set around failure slices

A useful test set represents decisions and risks, not just random questions. Each row should record the expected evidence and outcome at the level needed for its slice.

json 
{
  "case_id": "returns-policy-014",
  "slice": ["policy", "multi_document", "must_cite"],
  "question": "Can a damaged final-sale item be returned?",
  "expected_evidence": ["policy-v7#damaged-items", "policy-v7#final-sale"],
  "reference_answer": "Damaged items follow the defect process even when marked final sale.",
  "expected_behavior": "answer",
  "forbidden_sources": ["tenant-b/*"]
}

Include:

  • common questions weighted by real traffic;
  • rare but costly or regulated cases;
  • single-hop, multi-hop, ambiguous, and unanswerable questions;
  • recent, changed, deleted, and access-denied content;
  • typos, short queries, long conversations, and multiple languages;
  • known incidents and adversarial retrieval cases.

Keep a frozen core set for trend comparison and a rotating set for new traffic. Version documents, chunking, labels, and evaluator prompts so score changes remain explainable.

Evaluate retrieval before generation

Retrieval labels should answer whether the evidence needed for the question appears in the ranked results.

Context recall checks coverage: did the retriever find all required evidence? Context precision checks noise and ordering: did relevant evidence appear before irrelevant chunks?

Symptom Likely retrieval problem First experiments
Required source absent low recall query rewriting, hybrid search, filters, chunk boundaries
Correct source ranked late low precision/ranking reranker, scoring features, duplicate removal
Correct source denied authorization/index mismatch ACL propagation and policy tests
Old source returned freshness failure version filters, deletion and re-index checks
Too many redundant chunks diversity failure deduplication and source caps

Evaluate raw candidates and post-filter results separately. A retriever can have good offline recall while an authorization filter removes the only useful passage, or it can appear accurate because an evaluation corpus accidentally includes inaccessible data.

Evaluate generation against evidence

Answer faithfulness asks whether generated claims are supported by retrieved context. A common method extracts atomic claims and checks each one against evidence:

text 
faithfulness = supported_answer_claims / all_answer_claims

This score does not prove correctness. If a source is wrong, a faithful answer can repeat the error. If the model answers correctly from memory but ignores supplied evidence, it can be correct but unfaithful.

Also measure:

  • answer relevance: whether the response addresses the question;
  • completeness: whether required parts of a reference answer are present;
  • citation precision: whether each citation actually supports its nearby claim;
  • citation coverage: whether important claims have evidence;
  • abstention quality: whether the model declines when evidence is missing.

Do not reward verbosity. More claims create more opportunities for unsupported details and can hide a concise correct answer inside irrelevant prose.

Calibrate LLM judges

An LLM judge is a measurement instrument and needs calibration.

  1. Create a human-labeled calibration set with clear positive, negative, and borderline examples.
  2. Write a rubric that defines the evidence unit and allowed uncertainty.
  3. Blind the judge to system names and candidate ordering where possible.
  4. Compare judge labels with human labels by slice, not only overall.
  5. Review disagreements and update the rubric before changing thresholds.
  6. Re-run calibration when the judge model or prompt changes.

Use deterministic checks whenever possible:

typescript 
function deterministicChecks(result: {
  citedIds: string[];
  visibleIds: Set<string>;
  latencyMs: number;
  schemaValid: boolean;
}) {
  return {
    schemaValid: result.schemaValid,
    citationsVisible: result.citedIds.every((id) => result.visibleIds.has(id)),
    latencyBudgetMet: result.latencyMs <= 4000,
  };
}

Do not ask a judge to infer authorization, source identity, schema validity, or exact string constraints when code can check them directly.

Design release gates

Release gates should combine hard invariants, slice minimums, and change limits.

yaml 
hard_fail:
  unauthorized_retrieval: 0
  invalid_citation_ids: 0
  critical_cases_pass_rate: 1.0

slice_minimums:
  faithfulness:
    policy: 0.95
    general: 0.88
  context_recall:
    multi_hop: 0.85

regression_limits:
  task_success: -0.01
  p95_latency_ms: 500
  cost_per_success: 0.05

These numbers are illustrative, not universal. Set thresholds from labeled data, measurement variance, business impact, and the cost of false acceptance versus false rejection.

Use paired comparisons on the same cases. Report confidence intervals or repeated-run variation when model sampling or judge behavior is nondeterministic. A 0.01 average change is meaningless if run-to-run variation is 0.04.

Online evaluation and observability

Offline tests control inputs and enable release comparison. Online evaluation detects traffic drift, new documents, provider changes, and long-tail failures.

Sample by risk and novelty rather than uniformly:

  • always evaluate sensitive or consequential outcomes;
  • oversample new query clusters and changed sources;
  • inspect low-confidence retrieval and abstentions;
  • retain bounded, redacted evidence needed for review;
  • route user corrections and incidents into the regression set.

Keep evaluation separate from tracing. Traces explain what happened; evaluations judge whether it met a contract. The agent observability guide explains how to join those planes without collecting unnecessary private content.

A practical implementation sequence

  1. Define the decisions the RAG system is allowed to make.
  2. Build a versioned test schema and 5-10 meaningful slices.
  3. Label expected evidence before optimizing generation.
  4. Establish lexical/vector or hybrid retrieval baselines.
  5. Add deterministic policy, citation, and schema checks.
  6. Calibrate faithfulness and relevance judges.
  7. Set hard gates and per-slice regression limits.
  8. Run the suite in CI and before index or model changes.
  9. Sample production outcomes and add incidents to the suite.
  10. Review metrics quarterly for gaming and drift.

For retrieval tuning after diagnosis, see the hybrid search and reranking guide. For architecture fundamentals, use the RAG guide.

FAQ

Should retrieval and generation use the same test set?

They can share cases, but labels differ. Retrieval needs evidence relevance and ranking labels; generation needs claim support, reference outcomes, and abstention labels. Keep those fields separate.

Is context precision more important than context recall?

Neither dominates universally. Missing required evidence caps answer quality, while noisy context raises cost and can distract the generator. Optimize them under a context budget and by query slice.

Can user thumbs-up data replace evaluation labels?

No. Feedback is useful but affected by selection bias, presentation, user expertise, and delayed discovery of errors. Treat it as one online signal and investigate it with evidence.

Should production answers be sent to a judge synchronously?

Usually not for every low-risk request. Synchronous judging adds cost and latency and can create another dependency. Use deterministic inline checks, risk-based asynchronous sampling, and synchronous review only where the decision justifies it.

What should happen when metrics disagree?

Inspect the layer-specific evidence. High retrieval quality with low faithfulness points to generation. High faithfulness with low correctness points to bad sources or an incomplete reference. Good averages with a failed critical slice should block release.

Sources