What is Refusal Correctness?
Refusal Correctness is an evaluation property that measures whether a RAG system answers answerable requests and refuses unanswerable requests under a declared corpus, authorization scope, evidence-sufficiency rule, and response policy.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
Label answerability from the authorized corpus
The label depends on corpus revision, user and tenant permissions, query scope, freshness requirements, and the evidence-sufficiency rubric. A fact that exists somewhere on the internet does not make an enterprise query answerable. If authorized evidence supports only part of a multi-part request, define whether policy requires a bounded partial answer, clarification, escalation, or full refusal.
Measure both refusal error directions
Trust-Align distinguishes over-responsiveness on unanswerable questions from excessive refusal on answerable questions and evaluates grounded answers separately. This prevents a conservative system from looking safe solely because it refuses more often. Report correct answers, incorrect answers, correct refusals, false refusals, and over-responses by risk slice.
Build realistic unanswerable cases
UAEval4RAG defines six unanswerability categories and shows that one configuration does not consistently optimize answerable and unanswerable behavior across knowledge bases. Include missing entity, unsupported relation, temporal mismatch, ambiguous scope, access denial, and multi-hop evidence gaps that resemble the deployed corpus. Synthetic generation needs human review so impossible or accidentally answerable cases do not corrupt the denominator.
Key Characteristics
- Evaluates the answer-versus-refuse decision before answer quality
- Requires answerability labels bound to corpus and authorization revision
- Separates false refusal from unsafe over-response
- Treats incorrect answers as distinct from answerability errors
- Supports partial-answer, clarification, escalation, and full-refusal policies
- Needs risk-weighted slices because the two error directions have different costs
Common Use Cases
- Testing knowledge-base assistants on missing or inaccessible evidence
- Preventing a model from answering restricted questions from parametric memory
- Measuring whether stricter prompts cause excessive refusal
- Comparing retrieval and generation configurations on answerability behavior
- Defining release gates for high-impact over-response and false refusal
Example
Loading code...Frequently Asked Questions
What is a correct refusal in RAG?
It is a refusal issued when the authorized, current corpus lacks sufficient evidence for the requested claim scope. The evaluator must define answerability first; a generic refusal is not automatically correct.
What is the difference between false refusal and over-response?
A false refusal occurs when sufficient authorized evidence exists but the system declines. An over-response occurs when evidence is insufficient but the system answers anyway. The first harms usefulness; the second can create unsupported or unsafe claims.
Is a higher refusal rate safer?
No. A system can refuse every request and still be unusable. Report both error directions and weight them by consequence. High-impact unsupported answers may require a zero-tolerance gate, while false refusals may use a bounded rate and escalation path.
How should answerability be labeled?
Bind the label to a corpus revision, access scope, freshness rule, question interpretation, and evidence-sufficiency rubric. Review synthetic cases because an allegedly unanswerable question may be answerable from another authorized passage.
Should partial answers count as refusals?
Define the policy before scoring. A bounded partial answer can be correct when it states what is supported and explicitly refuses unsupported parts. Store per-part answerability so a single binary label does not hide mixed outcomes.