LLM Hallucination Detection: Evidence and Release Gates

Summary

LLM hallucination detection is an evidence problem, not a fluency test. A production system should decompose an answer into atomic claims, evaluate each claim against an authorized and versioned evidence snapshot, and then choose to release, revise, abstain, or escalate. A claim with insufficient evidence is not automatically false, and a consistent or well-formed answer is not automatically true.

Table of Contents

• Key Takeaways
• What Is an LLM Hallucination?
• Define the Truth Boundary First
• Build a Claim-Evidence-Decision Pipeline
• Implement a Deterministic Release Policy
• Why RAG Still Hallucinates
• What Common Controls Can and Cannot Prove
• Evaluate Detection and Abstention
• Separate Factuality from Tool Authorization
• Production Checklist
• FAQ
• Summary and Resources

Key Takeaways

• Evaluate atomic claims, not one opaque answer-level hallucination score.
• Pin the evidence corpus, revision, access scope, freshness rule, and citation anchors used for each decision.
• Use four verdicts: supported, contradicted, insufficient_evidence, and not_applicable.
• Treat consistency, model confidence, low temperature, and valid JSON as signals or controls, not truth guarantees.
• Evaluate retrieval, evidence quality, generation faithfulness, citations, abstention, and task success separately.
• Never let a factuality judge authorize a payment, deletion, message, or other external side effect.

What Is an LLM Hallucination?

An LLM hallucination is generated content that states an erroneous or false claim with apparent confidence. NIST AI 600-1 calls this risk confabulation and notes that it can mislead users. The operational definition still needs a declared boundary: false according to which source, revision, time, jurisdiction, or task contract?

Fluency is not part of the truth test. A hesitant sentence can be correct, while a polished answer with citations can be false. The following cases must not be collapsed into one label:

Unsupported and contradicted are different. Evidence may be missing because the corpus is incomplete, the question is out of scope, or the fact changed after the snapshot. Calling every unverified claim false corrupts both evaluation and user behavior.

Related failures are not synonyms

Instruction drift, arithmetic error, unsafe advice, bias, and policy refusal can coexist with hallucination, but each needs its own test. A response can be factually supported yet violate policy. It can also be faithful to supplied evidence while the evidence itself is wrong.

Two factuality axes are especially useful:

• Answer correctness: does the claim match the external truth standard for the task?
• Answer faithfulness: does the claim follow from the evidence supplied to the model?

A faithful summary of an outdated policy is not current-correct. An externally correct answer that is absent from the authorized evidence may still violate a closed-book support contract.

Define the Truth Boundary First

A hallucination test is only reproducible when the task contract defines what counts as evidence and when it is valid. Before generating an answer, record:

task:
  id: refund-policy-qa
  truthMode: closed-evidence
  asOf: 2026-08-10T09:00:00Z
  materialClaims: [eligibility, deadline, fee]

evidenceSnapshot:
  corpusId: support-policy
  revision: sha256:8c1f...
  authorizedTenant: tenant-42
  sourceAuthority: approved-policy
  freshnessRule: effective_at <= asOf < superseded_at
  citationUnit: source-id-and-section

failurePolicy:
  contradicted: block
  insufficientMaterialClaim: abstain
  invalidCitation: block
  reviewerRequiredFor: [legal-exception]

This contract prevents a verifier from silently searching a different web page, newer document, or unauthorized tenant corpus. It also makes disagreements diagnosable: the generator, retriever, evidence set, judge, and release policy each have a distinct identity.

Evidence needs lineage

Store the canonical source ID, source revision, authority, validity interval, access decision, retrieved span, and retrieval trace. Embeddings, summaries, graph edges, and reranker scores are derived aids; they are not substitutes for the source boundary.

When sources conflict, do not average them into a confident answer. Apply an explicit authority and recency policy, disclose the conflict when relevant, or escalate.

Build a Claim-Evidence-Decision Pipeline

The reliable unit of analysis is an atomic factual claim. FActScore demonstrated why long-form answers benefit from decomposition into atomic facts: one paragraph can contain both supported and unsupported statements.

1. Extract atomic claims

Split compound statements so each claim can receive one verdict. “The return window is 30 days and return shipping is free” contains at least two claims. Keep the character offsets that map claims back to the answer.

Claim extraction can itself fail by omitting material facts or turning hedged language into an absolute claim. Audit extraction recall on human-annotated answers, especially numbers, negation, temporal qualifiers, and citations.

2. Retrieve evidence under policy

Apply tenant and document authorization before ranking. Retrieval should return stable source anchors and revisions, not anonymous text fragments. A high similarity score says that text is nearby in embedding space; it does not say the text is authoritative or entails the claim.

3. Assign a bounded verdict

An NLI model or LLM judge can propose a verdict, but require a typed output and evidence IDs. Calibrate it against human labels, test position and verbosity bias, and preserve an insufficient_evidence path. Never parse free-form text with checks such as if "Supported" in response.

{
  "claimId": "claim-3",
  "claim": "The refund window is 30 days.",
  "material": true,
  "evidenceIds": ["returns-policy-v7#window"],
  "evidenceRevision": "returns-policy-v7",
  "verdict": "contradicted",
  "reasonCode": "VALUE_MISMATCH",
  "judgeVersion": "claim-judge-4",
  "reviewRequired": true
}

4. Aggregate by loss, not arbitrary weights

Do not assign “unable to verify” a universal half-error penalty. A contradicted dosage can be release-blocking; an unsupported optional anecdote can simply be removed. Aggregate decisions from claim materiality and task-specific loss.

Implement a Deterministic Release Policy

The following dependency-free Python fixture validates typed claim results and makes the final decision without trusting model prose or a self-reported confidence score:

from dataclasses import dataclass
from enum import Enum
from typing import Iterable


class Verdict(str, Enum):
    SUPPORTED = "supported"
    CONTRADICTED = "contradicted"
    INSUFFICIENT = "insufficient_evidence"
    NOT_APPLICABLE = "not_applicable"


@dataclass(frozen=True)
class ClaimResult:
    claim_id: str
    verdict: Verdict
    material: bool
    evidence_ids: tuple[str, ...] = ()
    citations_valid: bool = True


def release_decision(results: Iterable[ClaimResult]) -> str:
    claims = list(results)
    if not claims:
        return "review"

    if any(not claim.citations_valid for claim in claims):
        return "block"

    factual = [
        claim for claim in claims
        if claim.verdict is not Verdict.NOT_APPLICABLE
    ]
    if not factual:
        return "release"

    if any(
        claim.material and claim.verdict is Verdict.CONTRADICTED
        for claim in factual
    ):
        return "block"

    if any(
        claim.material and claim.verdict is Verdict.INSUFFICIENT
        for claim in factual
    ):
        return "abstain"

    if any(claim.verdict is Verdict.CONTRADICTED for claim in factual):
        return "revise"

    if any(claim.verdict is Verdict.INSUFFICIENT for claim in factual):
        return "revise"

    if any(
        claim.verdict is Verdict.SUPPORTED and not claim.evidence_ids
        for claim in factual
    ):
        return "review"

    return "release"


fixture = [
    ClaimResult(
        claim_id="refund-window",
        verdict=Verdict.SUPPORTED,
        material=True,
        evidence_ids=("returns-policy-v7#window",),
    ),
    ClaimResult(
        claim_id="return-fee",
        verdict=Verdict.INSUFFICIENT,
        material=True,
    ),
]

assert release_decision(fixture) == "abstain"
print(release_decision(fixture))
# Output: abstain

This policy is intentionally conservative, but not universal. A creative-writing product may mark nearly all content not_applicable; a medical or legal workflow needs stricter evidence, qualified review, and narrower automation. Version the policy and test it like application code.

Why RAG Still Hallucinates

Retrieval-Augmented Generation can provide current, private, and citable evidence, but retrieval is not a factuality proof. The original RAG paper reported improvements in its tested knowledge-intensive tasks; it did not establish that every production RAG pipeline eliminates hallucinations.

Treat RAG as a chain of fallible stages:

For implementation details, see Production RAG Evaluation and RAG Hallucination Mitigation.

What Common Controls Can and Cannot Prove

Many popular “anti-hallucination” techniques control one failure mode but are routinely overstated.

SelfCheckGPT provides evidence that sampling inconsistency can help black-box detection. Its result should remain a signal: consistent generations can repeat the same misconception. TruthfulQA likewise shows that models can reproduce common false beliefs and that scaling did not automatically improve truthfulness in its tested setting.

Lower temperature is therefore a reproducibility control. Use it when stable outputs help debugging or evaluation, but validate factuality independently.

Evaluate Detection and Abstention

A production evaluation must measure the detector, the answer, and the decision policy separately. Accuracy alone rewards guessing when abstention is allowed. OpenAI's analysis of why language models hallucinate highlights this incentive problem: an evaluation that only rewards correct answers can make guessing preferable to acknowledging uncertainty.

Build a versioned evaluation set

Include representative slices:

• answerable and deliberately unanswerable questions;
• current, stale, conflicting, and missing evidence;
• numbers, dates, negation, citations, and multi-claim answers;
• high-impact and low-impact claims;
• language, region, tenant, device, and user-permission slices;
• retrieval failures, prompt injection, timeouts, and partial dependencies.

Label atomic claims with evidence IDs and adjudication notes. Track inter-annotator disagreement instead of hiding ambiguous cases in a majority label.

Report metrics with denominators

Always report selective risk together with coverage. A system can achieve near-zero released-error rate by refusing everything, which is safe but useless. Plot risk against coverage and choose thresholds from business loss, not from a copied universal score.

If an LLM-as-Judge labels claims, compare it with qualified human labels, inspect confusion matrices by slice, and rerun calibration when the model, prompt, evidence format, or domain changes.

Separate Factuality from Tool Authorization

An answer verifier must never become an authorization system. Even a fully supported sentence does not prove that the requester may read a record, issue a refund, delete a file, or send a message.

For state-changing tools:

1. Authorize the principal, tenant, resource, and exact action with deterministic policy.
2. Validate typed arguments against current system state.
3. Require approval for high-impact operations.
4. Use idempotency keys and durable operation IDs.
5. Record succeeded, failed, or outcome_unknown; never infer success from model text.
6. Keep factuality, policy, safety, and execution-result gates independent.

This separation limits damage when an answer is wrong and also protects against prompt injection. Retrieved documents are evidence candidates, not instructions and not permissions.

Production Checklist

• Define the task's truth mode: open-world, closed-evidence, temporal snapshot, or domain authority.
• Version the model, prompt, retriever, corpus, evidence snapshot, judge, and release policy.
• Extract atomic claims and audit extraction recall.
• Keep contradicted separate from insufficient_evidence.
• Preserve canonical source anchors, revision, authority, access scope, and retrieval trace.
• Require citations to entail nearby claims; citation presence alone is insufficient.
• Add explicit revise, abstain, and qualified-review paths.
• Calibrate automated judges against human labels by risk slice.
• Gate releases on critical-slice regressions, not only aggregate averages.
• Measure selective risk, coverage, task success, latency, and cost.
• Keep tool authorization and external side effects outside the factuality judge.
• Log privacy-safe evidence and decision metadata for incident analysis.

FAQ

Why do LLMs hallucinate?

Autoregressive models optimize likely continuations, not a built-in database transaction against current truth. Training data can be sparse, conflicting, outdated, or contain popular misconceptions. Prompts and evaluations can also reward answering instead of abstaining. These mechanisms explain risk, but they do not predict one universal hallucination rate.

Can model confidence detect hallucinations?

Not by itself. Token probabilities describe likelihood under the model, and a model's verbal “90% confident” statement is uncalibrated unless tested against labeled outcomes. Confidence can support triage only after calibration on the same task and after checking calibration drift.

Is an unsupported claim false?

No. Insufficient_evidence means the declared snapshot cannot establish the claim. The claim may be true outside the corpus, newly changed, ambiguous, or simply irrelevant to the authorized source. Retrieve more evidence, abstain, or review it; do not silently convert absence of proof into contradiction.

Should two agreeing models be trusted?

Agreement is useful only as a bounded signal. Models can share training data, retrieval sources, prompts, and systematic misconceptions, so their errors are correlated. Verify material claims against independent evidence and measure whether agreement is calibrated on your task.

What is the best production hallucination metric?

There is no single best scalar. Use claim-level contradiction and insufficient-evidence metrics, citation precision and coverage, correctness, faithfulness, abstention precision and recall, selective risk, coverage, and end-to-end task success. Report results by risk slice.

Summary and Resources

Reliable LLM applications do not ask whether an answer “looks hallucinated.” They declare a truth boundary, preserve an evidence snapshot, evaluate atomic claims, and apply a deterministic decision policy. RAG, prompting, temperature, consistency checks, and automated judges can support this system, but none is a truth oracle.

Related resources:

• AI Hallucination glossary
• Grounding
• Answer Faithfulness
• Citation
• Production RAG Evaluation
• LLM-as-a-Judge Evaluation