GRPO: Learning from Groups of Responses

LLM Alignment: From PPO to DPO described PPO updates that use a learned value critic to estimate advantages. Group Relative Policy Optimization (GRPO) instead compares rewards among responses sampled for the same prompt. We derive this construction from DeepSeekMath, Section 4.1. We then examine the effects of subtracting the group mean, dividing by the group standard deviation, and using the resulting weights in a clipped update.

We begin with the baseline identity because replacing a critic by a sample group mean changes the dependence between a response and its update weight.

Recap: Which baselines preserve a reward gradient?

At a fixed prompt $x$, let $S_\theta(x,Y)=\nabla_\theta\log\pi_\theta(Y\mid x)$. For a fixed reward and a prompt-only baseline, the earlier score calculation gives

The baseline is fixed as the response varies. A mean computed from that same response group does not satisfy this independence argument directly; Section 2 calculates the resulting expectation.

1. One prompt, several sampled responses

Let $X\in\mathcal X$ be a prompt, including its task context, in a finite prompt set. Its distribution $\rho$ may be the empirical distribution over a fixed prompt dataset. For a verification task, any fixed answer key is used by the scorer, not supplied as an extra policy input. Let $\mathcal Y(x)$ be a finite set of valid token sequences, with an end-of-sequence token and a specified maximum horizon. Deterministic stopping/padding operations have no trainable log-probability contribution.

We distinguish three policies:

Assume differentiable positive probabilities on the same valid response support. For $G\ge2$, the group sampling law at iteration $k$ is

Thus responses are independent conditional on the prompt, not necessarily marginally. A batch consists of $B$ independent groups from $P_k^{(G)}$. Conditional on this batch, all sampled tokens, rewards, old-policy likelihoods, and computed advantages remain fixed during its optimizer updates.

Write $L_i=|Y_i|$ for the number of scored response tokens, $H_{i,t}=(X,Y_{i,<t})$ for a prefix, and

These response log-probabilities are sums, not length averages. An average over token losses, introduced later, is a separate weighting choice.

Let $r(x,y)\in\mathbb R$ be a bounded deterministic scorer, fixed throughout one rollout/update cycle, and set $R_i=r(X,Y_i)$. A learned reward model can supply $r$; a verifier can instead check an answer or run tests. GRPO does not require a learned reward model, but it still requires an informative reward.

In DeepSeek-R1-Zero, Sections 2.2.1–2.2.2, rule-based accuracy and format rewards supply this feedback. That is a particular reward design, not part of the mathematical definition of every GRPO run. A passed test need not certify every reasoning statement, and an imperfect verifier remains an imperfect training target.

Click to enlarge · Scroll sideways on small screens

2. What a group mean does to the reward gradient

To isolate the group baseline's effect, we first use full response scores without clipping, reward scaling, length averaging, or KL regularization. At a fixed prompt $x$, define the expected reward

Finite support permits differentiation under the sum. The likelihood-ratio identity and score identity give

As in the policy-gradient baseline derivation, subtracting a fixed prompt-only $b(x)$ preserves this gradient. A sample group mean requires more care because it includes the response whose score it multiplies.

A finite-group baseline identity

Proposition. Under the conditional iid law above, put $\bar R=G^{-1}\sum_jR_j$ and $S_i=S_{\theta_k}(x,Y_i)$. Then

To prove this, write $g_k(x)=\nabla_\theta V_\theta(x)|_{\theta_k}$. For $j\ne i$, conditional independence and the zero-score identity imply

Consequently,

Averaging over $i$ proves the claim. Averaging over $X\sim\rho$, with $\rho$ fixed, gives the same factor for $\nabla_\theta\mathbb E_{X\sim\rho}[V_\theta(X)]|_{\theta_k}$.

Next define the leave-one-out baseline

The other responses are conditionally independent of $Y_i$, so

This establishes an unbiased estimator at the rollout policy when we use the full response score without dividing by a group standard deviation. It does not show that every group baseline reduces variance. Standardizing the rewards or clipping the update can change the expectation.

3. Group-relative advantages are not just centering

For an explicit convention, use the divisor-$G$ standard deviation

With outcome supervision, the same $\widehat A_i$ is assigned to every scored token in response $i$. This is the construction in DeepSeekMath, Section 4.1.2. It is a group-relative weight, not automatically an estimate of the prefix-specific $Q^{\pi_k}(h,a)-V^{\pi_k}(h)$.

For a nonconstant group,

Using divisor $G-1$ instead multiplies these advantages by $\sqrt{(G-1)/G}$. Adding a stabilizer $\delta>0$ to the denominator is another specified variant: $(R_i-\bar R)/(s_R+\delta)$. It does not satisfy the unit-second-moment identity exactly. We use the explicit zero-variance branch above, so no undefined $0/0$ appears.

A binary-reward group

Suppose $R_i\in\{0,1\}$ and $C=\sum_iR_i$ counts successes. Conditional on $X=x$, let

For $0<C<G$, put $\widehat p=C/G$. Since $R_i^2=R_i$,

For example, $(R_1,R_2,R_3,R_4)=(1,0,1,1)$ gives

The sole failure receives a larger negative weight than each success's positive weight. For $C=0$ or $C=G$, every advantage is zero: there is no within-group reward contrast. This makes the reward part vanish, not necessarily a separately applied KL gradient.

The probability of a mixed-reward group is

Thus the supply of nonzero reward updates depends on both $G$ and the current success probability. It can be small for very hard or nearly solved prompts.

Why random normalization changes the gradient

The denominator $s_R$ depends on all rewards, including $R_i$. For this reason, the independence argument in Section 2 does not remove it from an expectation.

An exact counterexample uses one prompt and three responses with $\pi_\theta(j)=e^{\theta_j}/\sum_{\ell=1}^3e^{\theta_\ell}$, probabilities $\pi_k=(1/3,1/3,1/3)$, and rewards $(0,1,10)$. The ordinary reward gradient is

Take $G=2$ and let $e_j$ denote the $j$th coordinate vector. Distinct rewards give normalized advantages $(-1,+1)$ in reward order, regardless of the reward gap; equal responses give zero. Since $S_i=e_{Y_i}-\pi_k$, each unordered distinct pair contributes $(e_{\rm higher}-e_{\rm lower})/2$, and has probability $2/9$. Hence

The two vectors are not scalar multiples. Standardization can change the direction, not only the scale, of the mean-reward gradient. This calculation concerns the unclipped, full-sequence score estimator at $\theta_k$, without length averaging; it is not a convergence theorem for GRPO.

4. From group weights to clipped token updates

On a collected response define the current-to-old token ratio

For a batch $\mathcal B_k=\{(X_b,Y_{b,1:G},R_{b,1:G})\}_{b=1}^B$, write the reward surrogate as

The population surrogate averages the corresponding single-group expression under $P_k^{(G)}$. Because this is the old-policy sampling law, it is held fixed when we differentiate with respect to $\theta$. Only the current-policy probabilities inside the surrogate change.

The clipping mechanism can be read directly:

For positive advantage, the sample's incentive to increase its ratio flattens above $1+\varepsilon$. For negative advantage, its incentive to decrease the ratio flattens below $1-\varepsilon$. The sampled objective does not impose hard bounds on all token probabilities or guarantee improvement in expected reward.

Click to enlarge · Scroll sideways on small screens

At $\theta=\theta_k$, every ratio is one and lies in the unclipped interval. Differentiating the empirical reward surrogate gives

Two changes distinguish this update from the baseline proposition: dividing by a random group standard deviation and weighting by response length. When lengths vary, multiplying the sequence score by $1/L(Y)$ can prevent even a response-independent baseline from canceling. The reason is that $\mathbb E_\pi[S_\theta(X,Y)/L(Y)\mid X]$ need not be zero.

The old policy is also not the reference:

A single token ratio is not a full-response importance weight. Old-policy prefixes remain old-policy prefixes. Moreover, if generation uses a different temperature or truncation rule, its actual sampling law must be recorded; a nominal $\pi_k$ denominator does not by itself correct that mismatch. These are the same sampling distinctions used in the PPO–DPO article.

5. The KL term: value identity versus sampled gradient

For a fixed prefix $h$, abbreviate $p(a)=\pi_\theta(a\mid h)$ and $q(a)=\pi_{\rm ref}(a\mid h)$. Both are positive normalized token distributions. DeepSeekMath's equation (4) uses the nonnegative sampled quantity

To establish nonnegativity, set $z=q(a)/p(a)>0$ and apply $\log z\le z-1$. Equality holds at $z=1$. Next, when the token is sampled from $p$, its expectation is

This is a conditional token-KL value identity. If $A\sim p_k=\pi_k(\cdot\mid h)$ instead, generally $\mathbb E_{p_k}[k_\theta]\ne D_{\rm KL}(p\Vert q)$ after $\theta$ moves. For a fixed prefix and full support, the corrected identity is

Even an on-policy value identity is not automatically an unbiased gradient identity when differentiation treats samples as fixed. Let $s_\theta(a)=\nabla_\theta\log p(a)$. Direct differentiation gives

To differentiate its on-policy expectation, the sampling distribution contributes a score term:

The last equality uses $1-q/p+k_\theta=\log(p/q)$. Thus averaging the derivative of the sampled penalty need not equal differentiating its on-policy expectation. We recover the conditional forward-KL gradient by differentiating the corrected fixed-$p_k$ expression including its probability ratio. That correction still does not replace old-policy prefixes by current-policy prefixes.

For the concrete GRPO surrogate discussed here, define

This sampled regularization surrogate evaluates the penalty on fixed old-policy responses. Its expectation can therefore differ from forward KL under the current policy's trajectories. Averaging within each response also differs from full sequence KL. The latter sums token contributions and averages prefixes according to how often the current policy reaches them.

For the broader KL-regularized reward target and its exact optimizer, see the earlier objective-gap derivation. The centered, normalized, clipped surrogate above is a practical update construction, not an exact derivation of that optimizer.

6. A complete outcome-supervised update

Input: Initial policy $\pi_{\theta_0}$; reference $\pi_{\rm ref}$; prompt law $\rho$; fixed scorer $r$; group size $G\ge2$; batch size $B$; clipping parameter $\varepsilon$; penalty coefficient $\beta$; rollout rounds $K$; inner updates $M$; learning rates $\eta_{k,m}$.

Output: Updated policy $\pi_{\theta_K}$.

1. Freeze the scorer and reference. Specify the actual generation distribution, stopping rule, response-token masks, standard-deviation convention, and zero-variance handling.
2. For $k=0,\ldots,K-1$ do
3. Snapshot the rollout policy $\pi_k\leftarrow\pi_{\theta_k}$. Independently sample $X_b\sim\rho$ for $b=1,\ldots,B$.
4. For each $X_b$, sample $Y_{b,1},\ldots,Y_{b,G}\mid X_b\overset{\rm iid}{\sim}\pi_k(\cdot\mid X_b)$ and cache old-policy and reference token log-probabilities.
5. Score each completed response with $R_{b,i}=r(X_b,Y_{b,i})$. Compute $\bar R_b,s_{R,b}$ and $\widehat A_{b,i}$, using zero advantages when $s_{R,b}=0$.
6. Set $\theta^{(0)}\leftarrow\theta_k$. Keep all sampled tokens, rewards, cached probabilities, advantages and masks fixed.
7. For $m=0,\ldots,M-1$ do
8. Teacher-force the recorded responses under $\pi_{\theta^{(m)}}$. Sum or average only where specified in $\widehat L_{k,\rm GRPO}$; exclude prompt and deterministic padding tokens.
9. Compute token ratios and the clipped reward terms, then subtract the sampled reference penalty. Update $\theta^{(m+1)}=\theta^{(m)}+\eta_{k,m}\nabla_\theta\widehat L_{k,\rm GRPO}(\theta^{(m)})$.
10. End for. Set $\theta_{k+1}\leftarrow\theta^{(M)}$ and collect fresh groups for the next round.
11. End for.
12. Return $\pi_{\theta_K}$.

The pseudocode keeps the scorer and reference fixed to isolate the actor update; DeepSeekMath additionally considers outer iterations that refresh them. We display the full-batch gradient of the specified surrogate. Minibatch versions must retain the same group statistics and loss-reduction weights.

Zero reward variance does not require discarding a group in this algorithm: its reward term is zero, while its KL term can remain active. Filtering such groups is an additional design choice.

7. When rewards are assigned along a response

Outcome supervision attaches one score to the completed response. Process supervision instead supplies scores at selected reasoning-step endpoints. To state the construction in DeepSeekMath, Section 4.1.3, let $e_{i,j}$ be the final token of step $j$ in response $i$, for $j=1,\ldots,J_i$, and let $r_{i,j}$ be its process score.

Assume at least one step score is available in the group. Pool its $N_x=\sum_iJ_i>0$ step scores and define

using zero normalized scores if $s_{\rm proc}=0$. The token weight is then

Thus a token receives the normalized scores of reasoning steps ending at or after it. Unlike the outcome version, these weights vary within a response. This is still a specified group-normalized weighting rule; a learned process scorer, its normalization, and the chosen step segmentation do not automatically produce the true reward-to-go advantage.

8. DAPO: changing clipping, sampling, and token weights

Yu et al.'s DAPO changes how group-relative updates clip ratios, select groups, and average token losses. We examine each change in terms of the objective or sampling distribution it modifies.

Asymmetric clipping

Replace the symmetric interval by $[1-\varepsilon_{\rm low},1+\varepsilon_{\rm high}]$, with $0<\varepsilon_{\rm low}<1$ and $\varepsilon_{\rm high}>0$:

Choosing a larger upper threshold postpones the positive-advantage plateau. At a particular old token probability $p_k(a)$, that plateau begins at $p_\theta(a)=(1+\varepsilon_{\rm high})p_k(a)$, if reachable. It is a surrogate plateau, not a maximum admissible probability. Parameter sharing and other sample gradients can still move that probability further.

Dynamic sampling and its conditional distribution

For binary correctness rewards, keep only groups with $0<C<G$ and continue sampling until enough accepted groups have accumulated. With the acceptance probability $a_k(x)$ derived in Section 3, the accepted law is

provided the denominator is positive. Within a retained group, responses are no longer independent draws from the original response distribution conditional on the prompt. Retaining only mixed outcomes creates dependence between them. Prompts with $a_k(x)=0$ cannot enter. A practical rejection loop also needs a sampling budget and a rule for stopping if too few groups are accepted.

Filtering therefore maintains nonzero reward contrasts and changes the group/prompt distribution. It is not an unbiased sample from the original $\rho$ without further correction. Even without explicit filtering, normalized GRPO already gives zero reward contribution to constant groups.

Click to enlarge · Scroll sideways on small screens

The source comparison concerns training progress in its baseline setting. Its horizontal axis is update steps, not total generated tokens or wall-clock cost; rejected groups still require generation.

Token-level rather than response-level averaging

For a fixed retained batch, let $N=\sum_{b,i}L_{b,i}$ and abbreviate $f_{b,i,t}(\theta)=c_{\rm asym}(u_{b,i,t}(\theta),\widehat A_{b,i})$. Compare

In the first expression each response has total averaging weight $1/(BG)$. In the second it has $L_{b,i}/N$. Each token has equal averaging weight in the second, although its actual gradient still depends on its advantage, score, and clipping state. These reductions coincide for equal-length responses but generally differ otherwise.

DAPO's displayed objective uses token-level reduction, asymmetric clipping, and mixed-outcome filtering; its Section 2.3 also removes the reference KL term. We therefore do not present DAPO as merely GRPO with a larger upper clip.

Length-aware reward shaping

With maximum length $L_{\max}$ and buffer width $0<L_{\rm cache}\le L_{\max}$, DAPO's soft length penalty is

Adding this to a correctness reward changes what is rewarded; it is not just a numerical stabilization. Likewise, masking truncated responses changes which observed losses contribute. The binary acceptance calculation above applies to its stated binary reward model; it should not be reused unchanged for arbitrary composite rewards or masking schemes.

9. What follows from these calculations?

GRPO replaces a learned value critic with group comparisons; it does not remove rollout generation, reward design, or the need to control update size and sampling mismatch. The baseline identities explain one component exactly. They do not give a global convergence rate or guarantee that clipped neural optimization improves true task performance.

Compared with standard offline DPO, the core difference is data use: GRPO repeatedly generates groups and scores them, whereas fixed-dataset DPO optimizes recorded preference pairs. Both can use comparative information, but their empirical objectives and sampling laws differ.

The next note will examine sequence-level ratios and soft policy-update rules. Those methods change the update itself, so their likelihood ratios and guarantees must be derived separately.

References

1. Shao, Z., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Version 1, Sections 4.1.1–4.1.4, equations (3)–(4), Figure 4, and Algorithm 1. Main source for GRPO; the baseline identities and counterexample here are derived explicitly rather than attributed as the paper's theorem.
2. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Version 1, Sections 2.2.1–2.2.2. Source for the R1-Zero reward-design example; not used to identify token-level and full-sequence likelihood ratios.
3. Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms. Clipped surrogate and original Figure 1.
4. Yu, Q., et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. Version 1, Sections 2.3 and 3.1–3.4, equations (8)–(13), Figure 6. The accepted-distribution calculation is a derivation for the stated rejection-sampling model.