GRPO: Learning from Groups of Responses
Group-relative reward normalization, clipped token updates, and KL regularization, with baseline identities and DAPO extensions.
Blog
LLM Alignment: From PPO to DPO
Deriving KL-regularized alignment, PPO's rollout objective, and DPO's preference loss, with a population identification proof.
Coming soon
Beyond GRPO: Sequence Ratios and Soft Policy Updates
Sequence-level policy updates in GSPO and smooth adaptive gating in SAPO.