LLM Alignment: From PPO to DPO

Policy Gradients in Reinforcement Learning derived policy updates from sampled rewards. Language-model feedback often compares two responses instead of assigning a reward. We study two ways to learn from these comparisons: fit a reward model and optimize the policy with proximal policy optimization (PPO), or fit the policy directly with direct preference optimization (DPO).

The two approaches are connected by a reward objective that penalizes departure from a reference model using Kullback–Leibler (KL) divergence. We derive its optimal policy, use that policy to rewrite preference probabilities, and distinguish this population relationship from what either training procedure estimates.

Recap: Differentiating an expected reward

For a fixed prompt $x$ and reward $r$ that does not depend on the policy parameters, the score-function identity gives

The expectation changes with the policy. Section 4 applies this calculation to a reward with a parameter-dependent KL penalty, whose derivative must also be included. The finite-support assumptions below justify differentiation.

1. Prompts, responses, and three different policies

To keep the derivations finite, let $\mathcal X$ be a finite prompt set and $\mathcal Y_x$ a finite set of valid responses to prompt $x$. A finite vocabulary and maximum generation length $T$ suffice. A response includes its end-of-sequence token, with stopping forced by the horizon, and is padded after termination; padding is deterministic and contributes log-probability zero. We use undiscounted finite-episode returns.

For $y=(y_1,\ldots,y_T)$, an autoregressive language-model policy factors as

The prefix $h_t$ is the state, the next token is the action, and appending the token determines the next state. This is the finite-horizon interpretation from RL Foundations. Assume positive probabilities on the common valid-response support and differentiability in $\theta$.

Keep three policies distinct:

The reference and rollout policies may coincide initially but need not coincide later. All logarithms are natural; all response log-probabilities below are sums over generated tokens, not length averages.

The preference-data distribution

Fix an arbitrary ordering of $\mathcal Y_x$. A canonical pair $(U,V)$ has $U<V$, so its orientation is recorded before its preference label. Define the population sampling law

Here $C=1$ means $u$ is preferred to $v$. The preferred and rejected responses are $(Y_w,Y_l)=(U,V)$ when $C=1$, and $(V,U)$ otherwise. Together, the prompt, pair, and label draws define the comparison distribution $P_{\rm pref}$.

The pair distribution $Q_{\rm pair}$ describes how the data were collected, including their fixed ordering. It need not sample from the current policy. For example, the pairs may have been collected earlier from an SFT model.

For the empirical analysis, assume independent comparisons $O_i=(X_i,U_i,V_i,C_i)\overset{\mathrm{iid}}\sim P_{\rm pref}$. Write $\mathcal D=\{(x_i,y_{w,i},y_{l,i})\}_{i=1}^n$ for their realized, oriented form. An empirical average is over these fixed records, not newly sampled model responses. Repeated judgments or dependent annotators require a different sampling analysis.

2. From comparisons to a reward model

The Bradley–Terry model assumes a finite latent score $r^*:\{(x,y):y\in\mathcal Y_x\}\to\mathbb R$ such that

This is a modeling assumption about comparisons, not a consequence of language-model pretraining. We use unit logistic scale; any alternative preference-noise scale must be included explicitly or absorbed into $r^*$.

For a fitted reward model $r_\phi$, the conditional Bernoulli negative log-likelihood becomes

where $\Delta_{\phi,i}=r_\phi(x_i,u_i)-r_\phi(x_i,v_i)$. The equality follows by orienting each pair according to its label. Adding any prompt-only constant $c(x)$ to $r_\phi(x,y)$ leaves every comparison probability unchanged.

In reinforcement learning from human feedback (RLHF), this reward model can then score newly generated responses. Ouyang et al. (2022), Section 3.5 describe this SFT–reward-model–PPO pipeline. Their PPO-ptx variant also includes a pretraining-data term; the derivation here isolates the reward-minus-KL objective without that extra term.

Click to enlarge · Scroll sideways on small screens

3. Solve the KL-regularized reward problem

For now fix a reward function $r$, reference $q(y\mid x)>0$, coefficient $\beta>0$, and target prompt distribution $\rho$. Define

Here the inner expectation draws a response from the candidate policy, while the outer expectation draws a prompt from $\rho$. Neither distribution is automatically $P_{\rm pref}$. In PPO-based RLHF, $r$ is typically a reward model fitted before the policy update. For the population connection to DPO, we will instead use the latent $r^*$.

An exact optimizer and an exact objective gap

For each prompt define the partition function and exponentially tilted reference:

Finiteness of $\mathcal Y_x$ and $r$ ensures $0<Z_r(x)<\infty$. Normalization makes $\pi_r$ a positive probability distribution. Taking logarithms gives

Next substitute this identity into the prompt-specific objective $J_{r,x}$. For any response distribution $\pi$, with $0\log0=0$,

Nonnegativity of KL divergence establishes the optimizer and its gap:

Equality requires $\pi=\pi_r$ on prompts with positive $\rho$ probability. This is an unrestricted-policy result; a neural policy class may not contain $\pi_r$. The reward–policy transformation is developed in Rafailov et al. (2023), Section 4 and Appendix A.1.

What the reference does

For two responses $u,v$, the optimizer satisfies

The reward changes the reference odds rather than replacing them. For fixed $r,q$, $\pi_r\to q$ as $\beta\to\infty$; as $\beta\downarrow0$, mass concentrates on the reward maximizers, preserving their relative reference weights. A zero reference probability would exclude a response from any policy with finite forward KL. Positivity avoids that support obstruction in our derivations.

4. PPO: optimize the reward using rollouts

Evaluating the closed-form optimizer requires the sum over responses in $Z_r(x)$, whereas PPO updates a parameterized model from sampled responses. To connect that update to the objective, we first differentiate the regularized expected reward, then introduce the clipped surrogate.

The sequence-level score identity

Let $P_\theta$ denote the joint rollout law

Hold $r,q,\beta,\rho$ fixed while differentiating. Write

Differentiating both the sampling probabilities and the log penalty gives

The second line uses normalization, exactly as in the policy-gradient derivation:

Thus both the extra $-\beta$ term and any prompt-only baseline $b(x)$ disappear after expectation. Discrete token sampling does not make the expected objective nondifferentiable; the score identity differentiates its probabilities without differentiating through a sampled token.

Why the KL cost can be assigned to tokens

The autoregressive factorization yields

Let $d_{\pi,t}(\cdot\mid x)$ be the distribution of the prefix $H_t=(x,Y_{<t})$ generated by $Y\sim\pi(\cdot\mid x)$. The tower property gives the KL chain rule

An individual sampled log-ratio can be negative; its conditional expectation is a nonnegative KL divergence. The prefix law matters: averaging token KLs over unrelated fixed prefixes is not the sequence KL above.

At rollout iteration $k$, freeze a fitted reward $r_\phi$, reference $q$, and old policy $\pi_k$. A standard reward shaping choice on a sampled sequence is

It sums to $w_{\pi_k}(X,Y)$. The terminal score can equivalently be placed at the actual termination time before deterministic padding. The frozen shaped-reward policy gradient agrees with the regularized gradient at $\pi_k$, by the zero-score calculation above; away from that policy it is a local training construction, not the same objective for every $\theta$.

The clipped token update

Let $P_k$ be $X\sim\rho$, $Y\mid X\sim\pi_k(\cdot\mid X)$. The formulas assume responses are sampled from this policy. Changing the sampling temperature or removing tokens from its distribution changes that assumption.

Fit a value critic from the rollouts and obtain advantages $\widehat A_{k,t}$ for the shaped rewards. The critic targets the old policy's expected remaining return given $H_t$; the actor–critic note explains return and GAE estimators. In the expected surrogate below, we hold the critic fixed and treat each advantage estimate as a specified function of the rollout. During the actual batch update, we hold the recorded advantage estimates fixed.

The token likelihood ratio and clipped surrogate are

In implementation, the expectation is replaced by the rollout-batch average. Condition on that batch and its fitted advantages during the actor update. The critic, data collection, and subsequent rollout refresh are separate operations. This is the clipped actor objective of Schulman et al. (2017), Section 3, applied to the token process.

Click to enlarge · Scroll sideways on small screens

The old policy in $R_{\theta,k,t}$ is not the reference $q$. Clipping discourages particular profitable ratio changes relative to $\pi_k$; it does not impose a hard KL bound relative to $q$. Nor is a token ratio the whole-response ratio, which equals $\prod_t R_{\theta,k,t}$. These distinctions parallel the local-versus-joint clipping discussion.

5. DPO: eliminate the explicit reward model

Now use the population reward $r^*$ from the Bradley–Terry model and let $\pi^*=\pi_{r^*}$ be its unrestricted KL-regularized optimizer. The identity from Section 3 gives

Because a comparison uses the same prompt for both responses, subtracting their rewards cancels the partition term:

Consequently, the preference probability can be parameterized by a policy alone. For a candidate $\pi_\theta$ define the pair margin

Substitute $\sigma(m_\theta)$ into the conditional Bernoulli likelihood. Orienting each observed pair yields

This is the loss in Rafailov et al., equation (7) and Appendix A.2. It uses fixed preference pairs, with no newly sampled language-model rollout inside the standard DPO optimization loop. Collecting the original pairs may still have required generation and annotation.

Click to enlarge · Scroll sideways on small screens

For any positive normalized candidate policy $\pi$, the implicit reward

has $Z_{\widetilde r_\pi}(x)=\sum_y q(y\mid x)\pi(y\mid x)/q(y\mid x)=1$. Its regularized optimal policy is therefore exactly $\pi$. This representation is the reward-equivalence result underlying DPO; it does not say that an arbitrary learned $\widetilde r_\pi$ equals the true preference reward.

The DPO gradient

For one oriented pair, set $m=m_\theta(x,y_w,y_l)$ and $\ell=-\log\sigma(m)$. Since $d\ell/dm=\sigma(m)-1=-\sigma(-m)$ and $q$ is frozen,

Each sequence score is a token sum:

A gradient-descent update increases this pair's margin to first order by $\eta\sigma(-m)\|\nabla_\theta m\|_2^2$ for learning rate $\eta>0$. Poorly fitted or reversed comparisons receive more weight through $\sigma(-m)$. Shared parameters mean that other pairs' margins, and even a preferred response's absolute probability, need not all increase. If $\pi_\theta=q$ initially, every initial pair margin is zero and its weight is $1/2$.

6. When does population DPO identify the optimal policy?

Rewriting the reward gives a preference loss, but does not yet show that fitting it determines the optimal policy. That requires enough observed comparisons to identify relative response probabilities. The following finite-response result states the needed coverage conditions.

Proposition: identification from connected comparisons

Assume the finite setup of Section 1, $\beta>0$, positive $q$, and Bradley–Terry preferences generated by the finite reward $r^*$. Suppose:

• Every target prompt with $\rho(x)>0$ also has $\rho_{\rm pref}(x)>0$.
• At each such prompt, the undirected graph with response vertices and an edge $\{u,v\}$ whenever $Q_{\rm pair}(u,v\mid x)>0$ is connected.
• We minimize the population DPO risk over all positive response distributions, or over a class containing $\pi^*$, and a global minimum is attained.

Then every population minimizer equals $\pi^*(\cdot\mid x)$ on the target prompts. This is a population identification statement, not a finite-sample or optimization-rate guarantee.

The Bernoulli loss determines edge probabilities

First fix a prompt and a pair. Write $p=p^*(x,u,v)\in(0,1)$ and $s=\sigma(m_\pi(x,u,v))$. Averaging only over $C\sim\operatorname{Bernoulli}(p)$ gives

Subtract its value at the true probability:

with equality only at $s=p$. Let $\nu(x,u,v)=\rho_{\rm pref}(x)Q_{\rm pair}(u,v\mid x)$ be the prompt-pair law. Its population risk is

The policy $\pi^*$ realizes the true probability on every pair by Section 5. Consequently,

At a global minimum, the excess risk is zero. Each summand is nonnegative, so positive edge weights imply $\sigma(m_\pi)=p^*$ on every observed edge. Thus the loss is minimized only at the true preference probability for each compared pair. This property, called strict propriety, is also used in logistic regression.

Connectivity determines the remaining normalization

Because the sigmoid is one-to-one, matching the edge probabilities implies

on every edge. Connectivity propagates this equality along paths, so the common value is a single constant $c(x)$ across all responses. Thus

Finally, summing over $y$ forces $e^{c(x)/\beta}=1/Z_{r^*}(x)$, hence $\pi=\pi^*$. Prompt coverage transfers the conclusion to every target prompt. The proof requires no assumption that the collection policy equals the optimal policy, but it does require enough comparisons to identify all relative response scores.

What missing comparisons leave unidentified

If only responses $a$ and $b$ are compared while $c$ is never compared, take a uniform reference. The policies

have the same ratio $\pi(a)/\pi(b)=1$ and therefore the same predicted $a$-versus-$b$ preference probability $1/2$. No number of those comparisons identifies the mass assigned to $c$. Within a disconnected comparison graph, components can retain different normalizing constants.

Finite datasets introduce a further issue: a pair may have only one observed label even though its population preference probability is strictly between zero and one. Driving that empirical loss toward zero is not the same as recovering the population probability. With a restricted neural class, model misspecification, or incomplete optimization, the proposition's premises may also fail.

7. A two-response calculation

Fix one prompt with responses $a,b$ and reference probabilities $q(a)=1/5$, $q(b)=4/5$. Suppose

The Bradley–Terry reward difference is $r^*(a)-r^*(b)=\log3$. Choose the equivalent reward $r^*(a)=\log3$, $r^*(b)=0$. The regularized optimum is

Now recover the same policy from the population DPO loss. For $p=\pi(a)$, the margin and conditional risk are

The preceding Bernoulli argument minimizes this at $\sigma(m(p))=3/4$, hence

The 75% preference probability does not mean a 75% generation probability. The latter also depends on the reference and regularization coefficient. Here it is about 69.23%.

8. The DPO training procedure

Input: fixed preference dataset $\mathcal D$, frozen reference $q$, positive policy $\pi_{\theta_0}$, $\beta>0$, learning rates, and update budget $K$.

Output: fitted language-model policy $\pi_{\theta_K}$.

1. For each record, cache the reference response log-probabilities $\log q(y_w\mid x)$ and $\log q(y_l\mid x)$, or evaluate them with frozen reference parameters.
2. For $k=0,\ldots,K-1$:
3. Draw a minibatch of record indices uniformly from $\{1,\ldots,n\}$; do not generate new policy responses.
4. Evaluate $\log\pi_{\theta_k}(y_w\mid x)$ and $\log\pi_{\theta_k}(y_l\mid x)$ by teacher forcing on the recorded tokens.
5. Sum generated-token log-probabilities, including EOS; mask prompt and deterministic padding tokens.
6. Compute each reference-adjusted margin $m_{\theta_k}$ and the minibatch average of $-\log\sigma(m_{\theta_k})$.
7. Update $\theta_{k+1}=\theta_k-\eta_k\widehat\nabla_\theta L_k$, or use a specified gradient-based optimizer.
8. End for.
9. Return $\pi_{\theta_K}$.

Conditional on $\mathcal D$ and the current $\theta$, let $I\sim\operatorname{Unif}\{1,\ldots,n\}$ select a record independently of previous update randomness. For its loss $\ell_I$, uniform sampling gives

Averaging independently sampled indices preserves this equality. The gradient is unbiased for the empirical objective, not automatically for $J_{r^*}$; it does not imply that optimization finds a global population minimizer. No separate learned reward network or value critic is required by this standard DPO procedure.

9. What is shared, and what is different?

PPO is a general policy optimizer; DPO is a direct preference-learning construction, not a newer PPO clipping rule. Their shared population connection assumes the same preference reward, reference, and coefficient. Fitting $r_\phi$ with one restricted class and fitting $\pi_\theta$ by DPO with another need not yield identical fitted policies.

For a fixed reward, $\beta$ controls the exact optimizer's deviation from the reference, as Section 3 shows. In a finite-sample DPO run, however, changing $\beta$ changes both the classification margins and the learned implicit reward. The fixed-reward formula therefore does not guarantee monotone realized KL behavior for fitted DPO policies.

Finally, Bradley–Terry assumes a scalar reward can describe the comparisons. Annotator disagreement, systematic preference cycles, distribution shift and missing response coverage can challenge that model. Neither a lower empirical DPO loss nor a larger learned PPO reward is, by itself, a guarantee of truthfulness, safety, or improved human judgments outside the evaluated setting.

Continue with GRPO: Learning from Groups of Responses, which keeps rollout-based optimization but replaces the separately fitted value critic with group-relative comparisons of sampled rewards.

References

1. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Sections 3–5 and Appendices A.1–A.6; figure excerpt uses version 1.
2. Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. RLHF pipeline, Figure 2, Section 3.5 of version 1 and equation (2).
3. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. Clipped surrogate and original Figure 1.