What is Context Precision?
Context Precision is a ranking-aware RAG evaluation metric that rewards relevant retrieved contexts appearing before irrelevant ones, commonly computed as the mean of precision-at-rank values at ranks labeled relevant.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
Ranking-aware calculation
For binary relevance labels v_k at ranks 1 through K, compute CP@K = sum(Precision@k * v_k) / sum(v_k). Here Precision@k is the fraction of relevant items among the first k results. If no evaluated result is relevant, define the operational score explicitly, commonly zero, and report the no-relevant-result case rather than hiding it. This Average Precision-like protocol differs from plain precision@K.
Relevance labels define what the number means
Labels may come from gold passage IDs, human judgments, reference-answer support, or an LLM judge. These protocols are not interchangeable: document-level labels can credit a broad source, while claim-level labels require evidence for specific facts. Record the query set, corpus and index version, K, relevance rubric, judge version, and duplicate policy so runs can be compared.
Use precision with coverage and generation metrics
Lowering top-k can raise precision while deleting required evidence. Reranking can improve ordering without improving candidate recall, and duplicate relevant chunks can inflate apparent usefulness unless deduplicated. Diagnose context precision with context recall, answer faithfulness, task success, latency, and token use. Tune one stage at a time and inspect per-query labels before changing production thresholds.
Key Characteristics
- Rewards relevant contexts that appear earlier in the ranked result list
- Differs from the simple fraction of relevant results in top-k
- Depends on an explicit relevance rubric, K, corpus version, and duplicate policy
- Can use gold IDs, human labels, reference support, or calibrated judge labels
- Does not measure whether all required evidence was retrieved
- Supports diagnosis of candidate retrieval, reranking, filtering, and context assembly
Common Use Cases
- Comparing retrievers or rerankers on a fixed labeled query set
- Detecting relevant evidence buried below noisy top-ranked chunks
- Tuning top-k, filters, chunking, and query rewriting
- Monitoring retrieval regressions after corpus or embedding changes
- Reducing wasted context while preserving required evidence coverage
Example
Loading code...Frequently Asked Questions
Is Context Precision the same as precision at K?
Not in the common RAGAS definition. Plain precision@K is the relevant fraction among the first K results. Context Precision is ranking-aware: it averages precision-at-rank only at ranks judged relevant, so placing relevant evidence earlier raises the score.
What does a low Context Precision score mean?
It means relevant evidence is absent from the evaluated list or appears after too many irrelevant results. Inspect per-rank labels to distinguish weak candidate retrieval from poor reranking, broad chunks, duplicate evidence, or incorrect relevance judgments.
Can Context Precision improve while the RAG answer gets worse?
Yes. Reducing top-k may remove noise and raise precision but also remove a necessary fact, lowering context recall and answer completeness. The generator can also ignore excellent context, so retrieval and generation metrics must be read together.
How should relevance labels be created?
Prefer stable gold evidence IDs when the task permits them. Otherwise use a documented human or judge rubric that asks whether each context supports the query or reference claims. Calibrate automated labels against humans and preserve the labels for error analysis.
Can Context Precision scores from different systems be compared directly?
Only when they use the same query set, corpus version, K, relevance definition, granularity, duplicate handling, and judge. A score without that protocol is not portable, and changes in any of those inputs can move it without a true retrieval improvement.