RLHF Explained: Preference Data, Reward Models & PPO

TL;DR

RLHF (Reinforcement Learning from Human Feedback) turns human judgments into a training signal for a model policy. In the well-known InstructGPT recipe, teams first build an instruction-following policy, collect ranked responses, train a reward model, and optimize the policy with PPO while limiting drift from a reference model. The reward is still a proxy: higher reward does not prove that an answer is true, safe, or representative of every user. A production RLHF program therefore depends as much on data contracts, independent evaluation, and release controls as on the optimizer.

Key Takeaways

• RLHF is a family of methods for optimizing behavior from human feedback; SFT, reward modeling, and PPO are one influential implementation.
• A pairwise reward model learns the probability that reviewers prefer one response over another. It does not recover a universal human utility function.
• PPO clipping and the KL penalty solve different problems: clipping constrains an update relative to the rollout policy, while KL regularization discourages drift from a reference policy.
• Reward-model accuracy is necessary but insufficient. Calibration, disagreement, distribution shift, and reward overoptimization must be measured.
• Compare every candidate policy with an SFT baseline on independent capability, safety, factuality, and product slices before release.

What Is RLHF?

RLHF is a training approach that uses human judgments to construct a reward signal and then applies reinforcement learning to improve a model policy against that signal. It is valuable when the desired behavior is hard to encode directly but reviewers can make useful comparisons, such as choosing which of two answers is more relevant, appropriately cautious, or consistent with a product policy.

The phrase is often used too broadly. A precise vocabulary prevents architecture and evaluation mistakes:

The canonical three-stage pipeline is not the definition of RLHF. The 2017 paper Deep Reinforcement Learning from Human Preferences learned rewards from comparisons between trajectory segments in control tasks. Later language-model work, including Learning to Summarize from Human Feedback and InstructGPT, adapted the idea to text generation.

Most importantly, RLHF aligns a model to a sampled prompt distribution, a labeling policy, and a group of reviewers. The InstructGPT paper explicitly describes its result as alignment to the stated preferences of its labelers and researchers, not to a universal notion of human values.

The End-to-End RLHF System

An RLHF system is a chain of data, models, and evaluation decisions rather than one training job. Each boundary can change what the final reward means.

This view exposes four separate contracts:

1. Behavior contract: which prompts, users, languages, and failure cases matter.
2. Feedback contract: who judges responses, under which rubric, with which tie and escalation rules.
3. Optimization contract: which reward, reference policy, constraints, and stopping conditions guide updates.
4. Release contract: which independent metrics can block deployment and trigger rollback.

If any contract is implicit, a rising reward curve can hide a worse product.

Stage 0: Define the Target and Establish Baselines

Before collecting preferences, define the failure that preference optimization is expected to fix. "Make the model better" is not an evaluable objective.

A useful behavior specification names:

• the target prompt distribution and excluded use cases;
• the response dimensions reviewers should judge;
• precedence when dimensions conflict;
• acceptable refusal and uncertainty behavior;
• protected groups and high-impact scenarios;
• capability, safety, privacy, latency, and cost constraints.

Then measure at least a base model, a prompt-only baseline, and an SFT baseline. RLHF is justified only if these simpler interventions leave a measurable preference problem.

SFT commonly provides the initial instruction-following policy because raw next-token prediction does not reliably produce the interaction format reviewers need. It is not a mandatory theoretical stage. What matters is that the initial policy generates coherent candidates within the target domain and that the baseline remains frozen for comparison.

Stage 1: Build Preference Data as a Versioned Contract

Preference data records judgments between candidate responses, but the label alone is not enough. A usable record must preserve the context needed to interpret that judgment.

Pairwise Rankings Are Relative

Pairwise comparison is often easier to calibrate than an absolute score, but it only says that one response was preferred under one rubric and context. It does not say how much better it was, whether either response was acceptable, or whether another reviewer would agree.

Candidate generation also affects the learned preference. If all rejected responses are obviously broken, the model can learn superficial separation without learning difficult trade-offs. If one system always writes longer answers, the reward model may learn model identity or response length instead of quality. Randomize presentation order, mix generators, remove exact duplicates, and inspect simple features before training.

Disagreement Is Information

Do not collapse every disagreement into annotation noise. The 2026 revision of Diverging Preferences reports substantial disagreement in two preference datasets and attributes much of it to task underspecification, verbosity, response style, and refusal choices. Standard scalar reward models can turn a divided judgment into a confident score.

Track agreement by rubric dimension, prompt slice, language, and annotator cohort. Preserve ties and skip reasons. For subjective tasks, consider conditioning behavior on user preferences or routing ambiguous cases instead of training one global answer.

Split Before You Compare

Random row splits are unsafe when several comparisons come from the same prompt or conversation. Keep all candidates and derived pairs from one prompt in the same split. When possible, also hold out users, time periods, domains, and annotators.

This is not theoretical bookkeeping. InstructGPT showed labelers between four and nine responses for a prompt, creating up to K choose 2 correlated pairs. The paper grouped same-prompt comparisons during reward-model training because independently shuffling them caused rapid overfitting.

Stage 2: Train and Audit the Reward Model

A reward model maps a prompt-response pair to a scalar:

r_theta(x, y) -> real-valued score

A common Bradley-Terry preference model converts the score difference into a comparison probability:

P(y_chosen > y_rejected | x)
  = sigmoid(r_theta(x, y_chosen) - r_theta(x, y_rejected))

L_RM
  = -E[log sigmoid(r_theta(x, y_chosen) - r_theta(x, y_rejected))]

Only score differences identify pairwise preference. Adding the same constant to every reward leaves the probability unchanged, so an absolute score has no universal meaning unless the system defines and validates a calibration procedure.

What Reward-Model Metrics Actually Tell You

Evaluate against untouched human judgments from the intended deployment distribution. A reward model that separates old candidates may fail on responses generated by the policy it later trains.

A Runnable Go Audit

Deep-learning frameworks train the reward model, but a release pipeline still needs a framework-independent audit of frozen scores. This Go program computes pairwise accuracy, log loss, and Brier score overall and by slice. It fails on invalid scores and keeps the evaluation logic small enough to review.

package main

import (
	"fmt"
	"math"
	"os"
	"sort"
)

type pair struct {
	slice          string
	chosenReward   float64
	rejectedReward float64
}

type metrics struct {
	count   int
	correct int
	logLoss float64
	brier   float64
}

func softplus(x float64) float64 {
	switch {
	case x > 30:
		return x
	case x < -30:
		return math.Exp(x)
	default:
		return math.Log1p(math.Exp(x))
	}
}

func (m *metrics) add(chosenReward, rejectedReward float64) error {
	if math.IsNaN(chosenReward) || math.IsNaN(rejectedReward) ||
		math.IsInf(chosenReward, 0) || math.IsInf(rejectedReward, 0) {
		return fmt.Errorf("reward must be finite")
	}

	margin := chosenReward - rejectedReward
	probability := 1 / (1 + math.Exp(-margin))

	m.count++
	if margin > 0 {
		m.correct++
	}
	m.logLoss += softplus(-margin)
	m.brier += math.Pow(1-probability, 2)
	return nil
}

func (m metrics) report(name string) {
	fmt.Printf(
		"%s n=%d accuracy=%.1f%% log_loss=%.3f brier=%.3f\n",
		name,
		m.count,
		100*float64(m.correct)/float64(m.count),
		m.logLoss/float64(m.count),
		m.brier/float64(m.count),
	)
}

func main() {
	pairs := []pair{
		{slice: "coding", chosenReward: 2.4, rejectedReward: 0.3},
		{slice: "coding", chosenReward: 0.2, rejectedReward: 0.7},
		{slice: "coding", chosenReward: -0.1, rejectedReward: -0.4},
		{slice: "safety", chosenReward: 1.1, rejectedReward: 0.8},
		{slice: "safety", chosenReward: 0.4, rejectedReward: 1.2},
		{slice: "safety", chosenReward: 1.6, rejectedReward: 0.2},
	}

	overall := &metrics{}
	bySlice := map[string]*metrics{}

	for _, item := range pairs {
		if bySlice[item.slice] == nil {
			bySlice[item.slice] = &metrics{}
		}
		if err := overall.add(item.chosenReward, item.rejectedReward); err != nil {
			fmt.Fprintln(os.Stderr, err)
			os.Exit(1)
		}
		if err := bySlice[item.slice].add(item.chosenReward, item.rejectedReward); err != nil {
			fmt.Fprintln(os.Stderr, err)
			os.Exit(1)
		}
	}

	overall.report("overall")

	names := make([]string, 0, len(bySlice))
	for name := range bySlice {
		names = append(names, name)
	}
	sort.Strings(names)
	for _, name := range names {
		bySlice[name].report(name)
	}
}

Expected output:

overall n=6 accuracy=66.7% log_loss=0.598 brier=0.213
coding n=3 accuracy=66.7% log_loss=0.548 brier=0.193
safety n=3 accuracy=66.7% log_loss=0.649 brier=0.232

The slice results matter more than the overall average. A release should fail when a critical slice regresses even if a large easy slice keeps the aggregate metric high. Real evaluations should also attach confidence intervals and compare multiple seeds or checkpoints.

Where Bandit Methods Fit

At one decision boundary, selecting a response strategy or model variant and observing feedback only for the selected option resembles a Multi-Armed Bandit. When prompt, user, or request features change which option is best, the closer abstraction is a Contextual Bandit. Upper Confidence Bound and Thompson Sampling can be useful for explicit routing or experimentation choices, but they are not drop-in replacements for PPO: autoregressive generation has a vast structured action space, token-conditioned state, and delayed sequence-level reward.

Regret is useful for comparing exploration policies when a simulator or justified oracle exposes counterfactual value. Production logs usually do not reveal the reward of unchosen responses. Off-Policy Evaluation can estimate a frozen target policy from another policy's logs only when decision-time context, action propensities, outcome timing, and support are valid. Deterministic preference logs cannot establish arbitrary counterfactual policy value merely because they contain chosen and rejected text.

Stage 3: Optimize the Policy with PPO

PPO is an on-policy policy-gradient method: the current policy generates responses, the system scores those responses, estimates advantages, and updates the policy using the collected rollout batch.

An LLM implementation often has four logical components:

These are logical roles, not a mandatory four-model deployment topology. A framework may share a backbone, attach a value head, shard models, or offload components. Document the actual topology rather than multiplying parameter counts into a universal hardware requirement.

PPO Clipping Is Not the KL Penalty

Two constraints are often conflated:

1. PPO clipping limits the policy-ratio contribution for an update relative to the behavior policy that generated the rollout.
2. Reference-policy KL regularization penalizes behavior that moves away from a frozen reference, commonly the SFT policy.

A simplified clipped surrogate is:

L_clip(theta)
  = E[min(r_t(theta) * A_t,
          clip(r_t(theta), 1-epsilon, 1+epsilon) * A_t)]

r_t(theta)
  = pi_theta(a_t | s_t) / pi_old(a_t | s_t)

A common shaped reward adds a separate reference term:

R(x, y)
  = r_phi(x, y)
    - beta * sum_t(
        log pi_theta(y_t | x, y_<t)
        - log pi_ref(y_t | x, y_<t)
      )

The original PPO paper defines the clipped policy update. RLHF systems add reward-model scoring, reference-policy regularization, and language-model-specific rollout mechanics. The exact aggregation, KL estimator, value target, generation policy, and stale-rollout behavior are implementation choices that must be versioned.

The reward model usually provides a scalar for the completed response. It does not independently judge every token. Implementations may distribute the terminal reward across token-level returns and add per-token KL shaping before computing advantages.

Monitor the Optimization Path, Not Only the Endpoint

At each checkpoint, record:

• proxy reward and independent human or task metrics;
• KL divergence from the reference under a fixed measurement protocol;
• response length, entropy, truncation, and repetition;
• advantage mean and variance, value loss, and explained variance;
• PPO clip fraction, ratio distribution, and gradient norms;
• refusal, factuality, policy-compliance, and capability slices;
• rollout policy, reward model, tokenizer, prompt template, and code revisions.

Stop conditions should depend on independent quality gates, not on reward convergence. The policy can keep increasing proxy reward after real quality peaks.

Reward Overoptimization and Other Failure Modes

RLHF can fail whenever the measured preference differs from the behavior you actually need. The failure can enter before, during, or after training.

Scaling Laws for Reward Model Overoptimization demonstrated the core Goodhart failure in a controlled synthetic setup: optimizing a learned proxy initially improved a larger gold reward, then degraded it while the proxy continued to improve. The exact curves came from that setup, so they are evidence of the mechanism, not a universal stopping threshold.

KL regularization narrows policy movement but does not certify reward validity. A policy can exploit a local shortcut without traveling far under the chosen KL estimator, and an overly strong constraint can suppress useful improvement. The same principle applies to reward-model ensembles and uncertainty penalties: they can reduce particular failures, but only independent evaluation can establish whether the product improved.

What the InstructGPT Study Actually Established

InstructGPT is a landmark RLHF case study, not a universal production recipe. Its details are useful because they show how strongly results depend on data and evaluation design:

• The study recruited 40 contractors after a screening process.
• It used about 13,000 SFT training prompts, 33,000 reward-model training prompts, and 31,000 unlabeled PPO prompts. These are prompt counts, not interchangeable counts of pairwise comparisons.
• Reviewers ranked between four and nine responses per reward-model prompt, producing several correlated pairs from one ranking.
• A single 6B reward model was used for PPO policies at 1.3B, 6B, and 175B parameters.
• More than 96% of the prompt data was English.
• The main test distribution used prompts from held-out API customers, while separate analyses examined public NLP tasks and held-out labelers.

Under that study's prompts, labelers, models, and protocol, the 1.3B InstructGPT output was preferred to the 175B GPT-3 output. The paper also reported improvements in specific truthfulness and toxicity evaluations, along with remaining mistakes, no significant bias improvement on two benchmarks, and capability regressions that motivated mixing pretraining gradients into PPO.

Those findings do not establish a universal dataset size, model topology, hyperparameter set, or claim that RLHF makes every model safer. They demonstrate that a carefully measured preference-training pipeline can materially change behavior on its target distribution.

RLHF vs SFT, DPO, RLAIF, and RLVR

Choose an objective from the feedback you can trust and the behavior you need to change.

Read DPO vs RLHF for the objective derivation and method-selection matrix. The LLM fine-tuning guide covers the higher-level decision between prompt design, retrieval, SFT, parameter-efficient tuning, and preference optimization.

These methods can be combined, but more stages do not guarantee a better model. Gate each added stage on an independently measured failure and a matched baseline.

Production Evaluation and Release Gates

A credible RLHF evaluation separates the metric used for optimization from the metrics used to approve release.

Data Gates

• Version prompts, candidate policies, rubrics, annotator cohorts, and sampling weights.
• Report agreement, ties, skips, adjudication, duplicates, leakage, language coverage, and sensitive-data handling.
• Keep test prompts and annotators independent where practical.

Reward-Model Gates

• Compare against simple baselines such as response length and source model.
• Report pairwise accuracy, log loss, calibration, and confidence intervals by critical slice.
• Test newer policy outputs, adversarial pairs, reordered candidates, and counterfactual style changes.
• Reject a reward model whose aggregate gain hides safety or subgroup failure.

Policy Gates

• Compare base, SFT, and candidate policies under the same decoding settings.
• Use blind human comparisons plus task-specific deterministic metrics where available.
• Measure factuality, groundedness, refusal precision and recall, privacy, bias, robustness, and capability regression separately.
• Track response length and style so a superficial preference shortcut is visible.

Operational Gates

• Pin every model, dataset, tokenizer, prompt template, reward, and evaluation version.
• Canary the candidate, monitor distribution drift, and retain the previous policy for rollback.
• Treat user feedback as a new data source requiring consent, provenance, abuse resistance, and quality review.
• Keep runtime guardrails, authorization, and side-effect controls independent from model training.

RLHF changes probabilities. It cannot enforce tenant isolation, approve a payment, validate a medical fact, or guarantee that an agent action is authorized.

FAQ

What is RLHF?

RLHF uses human judgments to learn a reward signal and reinforcement learning to optimize a model policy against that signal. It is best suited to behavior that reviewers can compare consistently but that is difficult to express as a fixed objective.

Does RLHF always require SFT, a reward model, and PPO?

No. That sequence is the influential InstructGPT recipe. Other systems can start from a different policy, collect other forms of feedback, represent reward differently, or use another RL optimizer. DPO and related direct preference methods use similar comparison data but are normally distinguished from RLHF because they do not run a separate reward-model and RL loop.

What is the difference between RLHF and DPO?

PPO-based RLHF trains an explicit reward model and samples from the current policy during optimization. DPO applies an offline preference loss directly to chosen-rejected pairs under a reference-policy formulation. DPO usually reduces the live training components, but it does not remove annotation bias, distribution shift, overfitting, or the need for independent safety evaluation.

Does RLHF eliminate hallucinations or make a model safe?

No. Reviewers may prefer an answer that sounds confident without verifying it, and the reward model may amplify that shortcut. Evaluate factuality and policy compliance independently, require evidence where the product needs it, and enforce permissions and high-impact effects outside the model.

How much preference data does RLHF need?

There is no reliable universal threshold. Start with a representative pilot, keep a genuinely untouched evaluation set, and plot learning curves against an SFT baseline. Add data where disagreement, calibration, or slice failures show that coverage is insufficient; do not use record count alone as an alignment claim.

Summary

RLHF is a method for optimizing a model against a learned proxy of human judgment. Its central engineering challenge is not merely running PPO. Teams must define whose preferences count, preserve the conditions under which judgments were made, test whether the reward model transfers to new policy outputs, and stop optimization before proxy gains replace real quality.

Use SFT as a strong baseline, select RLHF only for a measured preference problem, and keep independent product, safety, and runtime controls in the release path.

Primary Sources

• Deep Reinforcement Learning from Human Preferences
• Proximal Policy Optimization Algorithms
• Learning to Summarize from Human Feedback
• Training Language Models to Follow Instructions with Human Feedback
• Scaling Laws for Reward Model Overoptimization
• Direct Preference Optimization
• Diverging Preferences: When Do Annotators Disagree and Do Models Know?
• Reinforcement Learning from Human Feedback