What is PPO?
PPO (Proximal Policy Optimization) is a family of on-policy policy-gradient methods that improves a policy from sampled trajectories while using a clipped or penalty-based surrogate objective to discourage excessively large updates.
Quick Facts
| Full Name | Proximal Policy Optimization |
|---|---|
| Specification | Official Specification |
How It Works
Clipped Surrogate Objective
The [original PPO paper](https://arxiv.org/abs/1707.06347) proposes repeated stochastic updates to a surrogate objective after collecting interaction data. PPO-Clip uses r = pi_new(a|s) / pi_old(a|s) and the minimum of the unclipped and clipped advantage-weighted terms. Clipping is asymmetric through the sign of the advantage: it removes incentives for an overly favorable change but does not discard every sample outside the interval. Value loss, entropy regularization, advantage estimation, masking and optimizer behavior are implementation choices around this core objective.
PPO in Language-Model Training
In classic RLHF, prompts produce on-policy response Tokens, a reward model or verifiable environment scores the trajectory, and a critic estimates returns so advantages can be computed. [InstructGPT](https://arxiv.org/abs/2203.02155) is an influential instance, not a universal PPO recipe. Modern systems also face rollout-policy staleness when generation and training are asynchronous. Record actor, rollout, reference, reward and tokenizer revisions; exclude Prompt and padding Tokens from assistant-action loss; and decide explicitly whether KL changes the reward or enters as a separate loss.
Diagnostics and Release Evidence
Training reward alone is insufficient because a policy can exploit the reward source. Monitor clipped fraction, policy and value losses, explained variance, entropy, approximate policy shift, reference divergence, reward components, response length, invalid outputs and rollout age. Evaluate the candidate and baseline on held-out prompts with deterministic checks, calibrated human or judge review, safety slices and repeated sampling. Stop or roll back when reward rises while task success, diversity, safety or out-of-distribution behavior regresses.
Key Characteristics
- Uses trajectories sampled from a recent behavior policy rather than a fixed offline dataset
- Reuses each rollout batch for multiple minibatch optimization epochs
- Weights updates with estimated advantages and current-to-old policy ratios
- Clips favorable surrogate changes but does not guarantee a strict trust region
- Often combines an actor with a critic, entropy control, and a separate reference-policy KL term
- Requires held-out outcome and safety evaluation because optimized reward can be exploited
Common Use Cases
- Optimizing an LLM from a learned reward model in a classic RLHF pipeline
- Training policies against deterministic code, math, game, or tool environments
- Improving multi-step agents when rewards depend on complete trajectories
- Studying online policy optimization with a learned value baseline
- Comparing actor-critic RL with DPO, GRPO, RLOO, or other post-training methods
Example
Loading code...Frequently Asked Questions
What exactly does PPO clipping constrain?
PPO-Clip limits the surrogate benefit of moving the sampled action-probability ratio outside a configured interval. It does not directly constrain every vocabulary probability, guarantee a KL bound, or make a destructive optimizer step impossible. Teams still monitor policy shift, gradients, value behavior and held-out outcomes.
Does PPO require a reward model?
No. PPO requires rewards or returns from an environment, not specifically a learned reward model. Classic RLHF often uses a reward model; RL with verifiable rewards can use tests, rules or environment outcomes. Any reward source can be incomplete or exploitable, so external evaluation remains necessary.
What is the difference between the old policy and the reference policy in LLM PPO?
The old or behavior policy generated the current rollout batch and appears in PPO's importance ratio. The reference policy is usually a frozen SFT checkpoint used to penalize behavioral drift. They can start equal but have different identities and purposes; combining their controls into one unnamed KL or clip setting causes implementation errors.
How are PPO, DPO, and GRPO different?
PPO is an online actor-critic family that learns from fresh trajectories and rewards. DPO is commonly an offline preference-pair objective with no rollout-and-critic loop. GRPO is an online policy method that estimates relative advantages from groups of outputs instead of a learned critic in its canonical form. Dataset, reward and evaluation quality still dominate the choice.
Which signals reveal a failing PPO run?
Warning signs include reward growth without held-out task gains, rising reference divergence, collapsing entropy, extreme clip fraction, poor value explained variance, length exploitation, invalid formats, stale rollouts and safety regressions. Compare against the unchanged baseline on identical prompts and stop before a high proxy reward is promoted as product improvement.