What is GRPO?
GRPO (Group Relative Policy Optimization) is an online policy-gradient method that estimates each sampled completion's advantage from rewards within a same-prompt group instead of training a separate Critic or Value Model.
Quick Facts
| Full Name | Group Relative Policy Optimization |
|---|---|
| Specification | Official Specification |
How It Works
For each prompt, a rollout policy generates a group of completions, and a Reward Function or Reward Model scores each completion. The original GRPO formulation centers each reward by the mean of that same group, scales it by the group's reward standard deviation, assigns the resulting completion-level advantage to its tokens, and applies a PPO-style clipped Policy Ratio; it can also penalize divergence from a Reference Policy. The group statistic is an empirical baseline, not the reward itself, and the original mean includes the current sample.
Removing the learned Critic reduces one major PPO component, but it does not remove rollout generation, reward design, or evaluation cost. GRPO is not synonymous with RLHF or RLVR: Human Feedback, AI Feedback, executable verifiers, rules, or learned scorers may supply rewards, and each has different failure modes.
Exact implementation identity matters. Original sample-level token averaging can introduce response-length bias, while within-group standard-deviation scaling can change the weight of prompts by reward variance. DAPO-style token aggregation, Dr. GRPO constant normalization, batch-level scaling, no scaling, and Leave-One-Out baselines are variants rather than interchangeable descriptions of the original objective. If every completion in a group receives the same reward, centered advantages are zero and that group supplies no relative learning signal. Policy-to-old-policy ratios used for clipped updates are also distinct from Importance Sampling correction for a separate inference engine whose Log Probabilities differ from the training engine.
A reproducible run therefore versions the policy, rollout policy, optional Reference Policy, Tokenizer, Chat Template, sampler, reward code, group construction, scaling, Loss aggregation, KL policy, truncation, and trainer. Release evidence must use reward-independent held-out metrics and inspect reward hacking, zero-variance groups, completion length, truncation, entropy, Clip Ratio, KL, sampler Log-prob gaps, safety, and capability regressions; increasing training reward alone is not a deployment result.
Key Characteristics
- Samples multiple completions for the same prompt and estimates relative advantages from their rewards without a learned Critic
- Uses a PPO-style Policy Ratio and clipping while treating Reference-Policy KL as an explicit, implementation-dependent choice
- Separates the Reward source from the optimizer, allowing verifiers, rules, learned models, or reviewed signals with different risks
- Requires an explicit baseline and Reward Scaling policy because group, batch, no-scaling, and Leave-One-Out variants weight prompts differently
- Makes Loss aggregation part of the algorithm identity because original GRPO, DAPO-style, and Dr. GRPO normalization have different length bias
- Needs rollout, reward, variance, length, entropy, clipping, sampler-consistency, safety, and held-out task diagnostics
Common Use Cases
- Improving mathematical or coding behavior with executable, independently validated outcome rewards
- Running online post-training where several diverse attempts can be sampled and scored for each prompt
- Comparing a critic-free group baseline with PPO, DPO, or other policy-optimization baselines
- Training bounded agent tasks with verifiable end states while separately testing long-horizon credit assignment
- Evaluating Reward, sampling, Loss, and KL variants under one immutable experiment and release contract
Example
Loading code...Frequently Asked Questions
How is GRPO different from PPO and DPO?
PPO usually learns a Critic to estimate advantages, while GRPO estimates a baseline from rewards for several completions of the same prompt. Both can use clipped online policy updates. DPO instead trains from fixed chosen and rejected pairs with a preference loss and no online rollout-and-reward loop. GRPO lowers Critic cost but adds grouped generation cost and does not make reward quality or stability automatic.
Does GRPO require a Reward Model or Reference Model?
GRPO requires a reward signal, not necessarily a learned Reward Model. Executable tests, exact-match checks, rules, human-derived scorers, or learned models can provide rewards. The original objective includes a Reference-Policy KL term, while some modern reasoning implementations disable it. Record the reward and KL policy explicitly instead of treating either implementation choice as part of the universal definition.
What happens when every completion in a GRPO group gets the same reward?
After subtracting the group mean, every relative advantage is zero, so the group supplies no relative policy-gradient signal. This is common when all attempts pass or all fail a binary verifier. Track the zero-standard-deviation rate by task slice; then improve prompt difficulty, exploration, rewards, or sampling rather than hiding the condition with a numerical epsilon.
Why can different GRPO implementations produce different results?
Implementations can differ in Reward Scaling, whether the baseline includes the current sample, token- or sequence-level Policy Ratios, Loss aggregation, KL, number of updates per rollout, truncation, external generation engines, and sampler correction. Original GRPO, DAPO-style Loss, and Dr. GRPO do not weight completion lengths identically. Version every choice and compare them under the same data and evaluation contract.
How should a GRPO-trained model be evaluated before release?
Compare it with the base, SFT, PPO or DPO, and reward-only selection baselines on immutable held-out tasks. Use metrics independent of the training reward, including task success, pass distributions, safety and capability regressions. Inspect reward hacking, zero-variance groups, completion length and truncation, entropy, clipping, KL when enabled, sampler Log-prob gaps, repeated-run uncertainty, serving parity, and rollback readiness.