What is Reward Model?
Reward Model is a learned model that maps a context and candidate behavior to a scalar or structured reward proxy, using preference, outcome, or process supervision to predict which behavior a specified evaluation policy would favor.
How It Works
Reward-model types describe different axes
Pairwise models learn relative preference between candidates, while pointwise models learn a label or rating for one candidate at a time. An Outcome Reward Model (ORM) scores a completed result; a Process Reward Model (PRM) supervises intermediate reasoning steps. A Generative Reward Model emits an assessment or critique rather than only one scalar. These categories are not mutually exclusive: supervision format, scoring granularity, and output architecture are separate design choices.
Bradley-Terry training learns score differences
For prompt x, preferred response y_w, and rejected response y_l, a common model sets P(y_w > y_l | x) = sigmoid(r(x,y_w) - r(x,y_l)). The negative log-likelihood is softplus(-(r_w - r_l)). Only the difference affects this objective, so adding the same constant to both scores leaves the probability and loss unchanged. Raw reward zero has no intrinsic meaning, and scores from different checkpoints are not comparable without shared anchors or calibration.
The data contract determines what the score means
Each record should preserve the prompt and conversation state, candidate texts and generator versions, rubric version, reviewer cohort, choice or tie, confidence, presentation order, provenance, consent, and license. Keep all pairs derived from one ranking in the same split, and group related conversations, users, templates, or source documents when they can leak. InstructGPT grouped comparisons from the same prompt during training because treating correlated pairs as independent caused rapid overfitting.
Evaluation must separate ranking, probability, and product value
Held-out pairwise accuracy measures whether the preferred candidate ranks higher; log loss and Brier score evaluate the predicted pairwise probability, while reliability bins reveal calibration. Also inspect margins, ties, reviewer disagreement, and correlations with length, formatting, refusal rate, or generator identity. Report language, domain, safety, and difficulty slices. Finally, test unseen prompts and downstream Best-of-N or policy optimization against independent human, task, and safety outcomes.
Policy optimization creates its own distribution shift
Once a policy is optimized against the reward model, it produces outputs selected to score well under that model rather than ordinary training candidates. Proxy reward can keep increasing while independent quality falls, a form of reward overoptimization. Ensembles and uncertainty estimates can expose some disagreement but cannot remove shared shortcuts. Version the model, tokenizer, prompt template, normalization, dataset, rubric, and evaluation suite; stop or roll back when reward-independent gates regress.
Reward models are not every kind of evaluator
Reward function is the broader term and may be a deterministic rule, executable test, environment outcome, or learned model. A verifier judges a property against evidence such as executable tests, formal checks, or a learned correctness signal; when an outcome is objectively checkable, it need not model human preference. LLM-as-a-Judge uses a language model at inference time to rank, score, or critique outputs and can create labels without being a separately trained reward model. DPO optimizes a policy directly from preference pairs and does not require a separate explicit reward model or online reward-scoring loop.
Key Characteristics
- Learns a rubric- and population-dependent proxy rather than a universal utility or truth score
- May use pairwise, pointwise, outcome, process, or generative supervision
- Pairwise Bradley-Terry objectives identify score differences, not an absolute score origin
- Can rank fixed candidates well yet fail on outputs from a newly optimized policy
- Is vulnerable to label bias, leakage, spurious features, distribution shift, and reward hacking
- Requires reward-independent evaluation, versioned artifacts, monitoring, and rollback criteria
Common Use Cases
- Supplying a learned reward signal for PPO-based RLHF or another policy optimizer
- Selecting the best response from several candidates in Best-of-N inference
- Scoring complete answers with an ORM or intermediate reasoning steps with a PRM
- Filtering or prioritizing generated examples for human review and dataset curation
- Monitoring whether a model version still matches a versioned product rubric
Example
Loading code...Frequently Asked Questions
How is a reward model trained?
A common LLM reward model receives a prompt plus candidate response and learns from chosen-rejected comparisons. Under a Bradley-Terry objective, it raises the score difference for the preferred answer. Other designs learn pointwise ratings, final outcomes, step labels, or generated critiques, so the training target must be stated explicitly.
What is the difference between an Outcome Reward Model and a Process Reward Model?
An Outcome Reward Model scores the completed answer or trajectory. A Process Reward Model evaluates intermediate steps, which can provide denser supervision and reveal where reasoning fails. A PRM needs trustworthy step-level labels and does not guarantee that the final result is correct.
Is a reward model the same as an LLM judge or verifier?
No. LLM-as-a-Judge is an inference-time evaluation pattern and may be prompt based. A verifier judges against evidence such as executable tests, formal checks, or a learned correctness signal. Either can provide labels or rewards, but neither is automatically a separately trained preference model.
How should reward models be evaluated?
Use prompt-grouped held-out data and report pairwise accuracy, log loss, Brier score or calibration bins, margins, ties, disagreement, and domain or safety slices. Then test unseen prompts and optimized-policy outputs against independent human, task, and safety measures. Raw scores from different checkpoints should not be compared without anchors.
Why can a higher reward-model score produce a worse model?
Optimization searches for outputs that exploit whatever the proxy learned, including length, style, confident wording, or other shortcuts. The policy-induced distribution can differ from reward-model training data, so proxy reward may rise while independent quality falls. Bound optimization with held-out gates, fresh human labels, monitoring, and rollback.