What is Answer Faithfulness?
Updated on
Answer Faithfulness is a RAG evaluation property that measures the proportion of factual answer claims supported by the exact context supplied to the generator under a declared claim-decomposition and entailment protocol.
Quick Facts
| Created | Formalized for RAG evaluation in the RAGAS framework in 2023 |
|---|---|
| Specification | Official Specification |
How It Works
Define the scoring unit before the threshold
A reproducible evaluator states what counts as a factual claim, how compound sentences are split, whether qualifiers and numerical conditions remain attached, what evidence scope the judge can inspect, and how abstentions or empty answers are handled. A simple score is supported claims divided by evaluated claims. Without a stable decomposition policy, two evaluators can assign different denominators to the same answer.
Faithfulness is not correctness or citation quality
A faithful answer can repeat an outdated source, while a factually correct answer can be unfaithful if the fact came from model memory rather than supplied evidence. Citation presence is also insufficient: a link may be attached to a claim it does not entail, and uncited claims affect citation completeness. Evaluate source quality, retrieval coverage, answer correctness, citation support, and task usefulness as separate properties.
Calibrate judges before using release gates
Human reviewers, natural-language-inference models, and LLM judges can label claim support, but ambiguous wording, long contexts, prompt order, and judge self-preference introduce error. Build a stratified human-labeled set, measure agreement and false-negative or false-positive patterns, pin evaluator versions, and inspect score distributions by query class. A release gate should combine faithfulness with minimum sample size and critical unsupported-claim rules.
Key Characteristics
- Scores answer claims against supplied evidence rather than model confidence
- Uses an explicit claim unit and support decision to make the denominator inspectable
- Separates generation grounding from retrieval coverage and source truth
- Can use human review, entailment models, or calibrated LLM judges
- Is sensitive to claim decomposition, evidence scope, and evaluator version
- Should surface unsupported claims, not only a corpus-level average
Common Use Cases
- Detecting unsupported factual additions in RAG answers
- Comparing prompts or models while holding retrieved context fixed
- Defining release gates for evidence-grounded assistants
- Sampling production traces for claim-level review
- Separating generator regressions from retriever failures
Example
Loading code...Frequently Asked Questions
Is answer faithfulness the same as factual correctness?
No. Faithfulness asks whether claims follow from the context shown to the generator. Correctness asks whether claims agree with a trusted reference or the real world. A complete evaluation needs both when source truth matters.
How is answer faithfulness calculated?
A typical protocol extracts atomic factual claims, labels each as supported or unsupported by the supplied context, and computes supported claims divided by all evaluated claims. The report should retain claim-level labels because an average hides which statement failed.
How should answers with no factual claims be scored?
Do not silently assign a perfect score. Mark the metric as not applicable or route the answer to a separately defined abstention or empty-answer policy. Otherwise systems can improve the average by producing less useful, noncommittal text.
Can an LLM judge reliably measure faithfulness?
It can scale evaluation, but its output is an estimate. Calibrate it against representative human labels, pin the judge prompt and model version, monitor agreement by query type, and manually review safety-critical claims and threshold-adjacent failures.
Does a high faithfulness score mean a RAG system is good?
Not by itself. A system can faithfully answer from incomplete or false evidence, omit important facts, or fail the user's task. Pair faithfulness with context recall and precision, source-quality checks, answer correctness or task success, citation metrics, latency, and cost.