LLM-as-a-Judge Beyond ROUGE and BLEU: A Calibrated Evaluation Method

The Boundary

ROUGE, BLEU, and exact match compare a candidate with a reference under a defined string or token rule. An LLM judge estimates whether a candidate satisfies a rubric. Neither is a universal measure of usefulness, truth, safety, or authorization.

Use the smallest evaluator that answers the question:

An evaluator is a measurement instrument. It needs a construct, a protocol, known failure modes, and calibration.

Where ROUGE and BLEU Still Help

Lexical metrics are not obsolete. They are useful when surface overlap is part of the contract:

• translation regression with comparable references;
• extractive summarization;
• keyword or entity extraction;
• templated outputs;
• detecting an accidental change in a stable format.

They become weak when a task permits multiple valid phrasings, requires factual verification, or has several quality dimensions. Report them as one slice, not as a complete quality score.

Define the Construct First

Before writing a judge prompt, define:

1. the user or business outcome;
2. the evidence the evaluator may use;
3. required facts or fields;
4. forbidden claims and actions;
5. acceptable alternatives;
6. abstention or escalation behavior;
7. the authoritative oracle, if one exists.

A rubric should describe observable behavior, not vague “intelligence”:

Correctness:
  pass: every material claim is supported by the supplied evidence or an approved source
  fail: any material claim contradicts evidence or invents a source

Instruction following:
  pass: required fields are present and constraints are respected
  fail: a required field is missing or a forbidden action is proposed

Evidence quality:
  pass: each high-impact claim maps to a cited evidence span
  review: evidence is incomplete or the claim is ambiguous

Do not use the rubric as an authorization policy. Server-side policy remains authoritative.

Three Judge Modes

Pointwise

Evaluate one response against a rubric. Use for sampled monitoring and regression slices. It is easy to operationalize but absolute scores drift when the judge or rubric changes.

Pairwise

Compare two responses for the same case. Use for controlled A/B experiments. Pairwise results are relative, not an absolute quality guarantee; randomize order and allow ties.

Reference or Evidence Guided

Compare claims with a reference, source document, expected structure, or execution result. This works best when the reference is not treated as the only valid wording and when unsupported additions are detected separately.

Keep judge instructions and evidence separate from candidate content. Candidate text can contain prompt injection or instructions aimed at the evaluator.

A Safe Output Contract

Ask for a small structured verdict, not hidden chain-of-thought:

{
  "verdict": "pass|fail|review",
  "dimension_scores": {
    "correctness": 0,
    "evidence_support": 0
  },
  "supported_claim_ids": ["c1"],
  "unsupported_claim_ids": ["c2"],
  "uncertainty": "low|medium|high",
  "reason_codes": ["missing_evidence"]
}

Validate the JSON with a parser, reject unknown values, cap reason length, and treat malformed output as an evaluation error. A judge’s explanation is evidence for debugging, not a trusted proof.

Calibrate the Judge

Position and Order

For pairwise tests, randomize A/B order and rerun a sample with the order swapped:

from dataclasses import dataclass
from typing import Callable

@dataclass(frozen=True)
class PairVerdict:
    winner: str  # "A", "B", "TIE", or "REVIEW"

def reconcile(
    first: PairVerdict,
    swapped: PairVerdict,
) -> str:
    if first.winner == "A" and swapped.winner == "B":
        return "A"
    if first.winner == "B" and swapped.winner == "A":
        return "B"
    if first.winner == swapped.winner and first.winner in {"A", "B", "TIE"}:
        return first.winner
    return "REVIEW"

This detects order sensitivity; it does not eliminate every bias.

Human Anchors and Controls

Maintain a control set with human-reviewed cases covering easy, ambiguous, adversarial, and high-risk examples. Compare judge decisions with human labels by task slice. Record disagreement and confidence, and recalibrate when the model, prompt, rubric, parser, or data distribution changes.

Do not publish a universal agreement threshold. The acceptable error depends on harm, reversibility, and the cost of human review.

Biases to Test

• verbosity and style preference;
• position and order;
• self-preference for the judge’s model family;
• agreement with a reference even when the reference is wrong;
• sensitivity to formatting or identity cues;
• refusal to mark insufficient evidence;
• prompt injection inside candidate or retrieved content.

Blind irrelevant metadata, keep candidate order randomized, and include concise correct answers in the control set.

RAG Evaluation

Evaluate retrieval and generation separately:

Do not ask a judge to use general knowledge when measuring faithfulness. Give it the question, bounded evidence, answer, and claim identifiers. A correct fact unsupported by retrieved context may still be a retrieval failure.

Deterministic and Human Oracles

Use deterministic checks for code execution, schemas, arithmetic, policy decisions, permissions, and side effects. Use human review for ambiguous, high-impact, or novel cases. Use a judge for scalable triage and comparison where its calibration evidence supports the use.

An ensemble does not automatically create truth. Multiple judges can share the same bias. If they disagree, preserve the disagreement or escalate instead of manufacturing a precise average.

Cost, Privacy, and Sampling

Judge input often contains user questions, retrieved documents, and model responses. Before sending it to a provider:

• minimize and redact personal or confidential data;
• record purpose, retention, residency, and deletion behavior;
• hash or surrogate identifiers;
• cap context and explanation size;
• avoid storing raw candidate text in default telemetry;
• distinguish estimated cost from provider billing.

Choose sampling by risk and statistical purpose. A fixed “evaluate every request” or “sample X percent” rule is not universally correct. High-impact cases may require complete metadata and human review, while low-risk traffic may use stratified samples.

CI/CD Release Gates

Version all of the following:

task set + data digest + prompt + rubric + judge model
parser + oracle + sampling + environment + report

Use a gate such as:

contract checks
  -> deterministic scenarios
  -> replay comparison
  -> abuse and privacy cases
  -> calibrated judge triage
  -> human review of disagreement
  -> cost/latency check
  -> canary and rollback

Gates should express invariants and workload budgets. Do not block a release solely because an arbitrary judge average moved by a fixed amount without uncertainty and slice analysis.

Common Failure Modes

• treating a judge score as ground truth;
• forcing every answer into one reference;
• asking for hidden reasoning as an audit log;
• allowing candidate content to control the judge prompt;
• parsing model JSON without schema validation;
• comparing models with different prompts, tools, or retrieval context;
• averaging incompatible dimensions into one number;
• retaining raw evaluation data without deletion and access controls;
• using ROUGE/BLEU to score factual or policy correctness;
• using a judge to authorize money movement, access, deletion, or external writes.

Practical Checklist

• [ ] Define the construct, evidence, acceptable alternatives, and oracle.
• [ ] Use exact or execution checks when the answer is objectively verifiable.
• [ ] Keep judge output small, structured, bounded, and schema-validated.
• [ ] Randomize pair order and include ties and insufficient-evidence outcomes.
• [ ] Calibrate against human control cases by risk and task slice.
• [ ] Test verbosity, self-preference, formatting, identity, and prompt injection bias.
• [ ] Separate retrieval, evidence support, answer correctness, and safety.
• [ ] Redact, minimize, sample, retain, and delete evaluation data deliberately.
• [ ] Version task sets, rubrics, prompts, judges, parsers, oracles, and reports.
• [ ] Combine judge evidence with abuse tests, cost/latency budgets, approval, canary, and rollback.

Conclusion

LLM-as-a-Judge is useful when it is treated as a calibrated, fallible measurement instrument. ROUGE and BLEU remain valuable for the contracts they actually measure; deterministic and human oracles remain authoritative where they apply. A mature evaluation system makes uncertainty visible, protects evaluation data, and refuses to turn a fluent model verdict into proof of correctness, safety, or permission.

Primary Sources

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• G-Eval: NLG Evaluation using GPT-4
• FAVA: Fine-grained AI-generated Text Factuality Evaluation
• HELM: Holistic Evaluation of Language Models
• NIST AI Risk Management Framework