Language Model Architectures
Shared KV Memory: From Transformers to DeepSeek
Causal attention, KV caching, and the memory savings from sharing context across Transformer layers.
Blog
LLM Post-Training
GRPO: Learning from Groups of Responses
Group-relative reward normalization, clipped token updates, and KL regularization, with baseline identities and DAPO extensions.
LLM Post-Training
LLM Alignment: From PPO to DPO
Deriving KL-regularized alignment, PPO's rollout objective, and DPO's preference loss, with a population identification proof.
Multi-Agent Reinforcement Learning
MARL, Part 4: Multi-Agent Policy Gradients
Joint policy gradients, deterministic actor–critic updates, and the sequential importance corrections behind HAPPO.
Multi-Agent Reinforcement Learning
MARL, Part 3: Centralized Training, Decentralized Decisions
Local information constraints, centralized critics, and why an agent’s memory matters even when training observes the state.
Multi-Agent Reinforcement Learning
MARL, Part 2: Cooperation and Credit Assignment
Shared rewards, counterfactual policy gradients, and additive or monotonic value decomposition for cooperative agents.
Multi-Agent Reinforcement Learning
MARL, Part 1: Foundations and Markov Games
Joint policies, best responses, and Nash equilibria, with a minimax Bellman equation and contraction proof for zero-sum Markov games.
Policy Optimization
Policy Gradients in Reinforcement Learning
Deriving the policy gradient and comparing return-based, actor–critic, natural, trust-region, and deterministic methods.
RL Foundations
Reinforcement Learning, Part ii: Partial Observability, Offline Data, and DTRs
What changes with hidden states or fixed datasets, and how history-based offline learning connects to DTRs.
RL Foundations
Reinforcement Learning, Part i: Foundations and Learning Methods
MDP assumptions, Bellman equations, a value-iteration proof, and the main families of RL methods.
Coming soon
Policy Optimization
Simultaneous Direct Policy Learning: The Policy-Gradient Connection
A forthcoming note on direct policy learning and policy gradients.
Coming soon
LLM Post-Training
Beyond GRPO: Sequence Ratios and Soft Policy Updates
Sequence-level policy updates in GSPO and smooth adaptive gating in SAPO.
Coming soon
LLM Training & Efficiency
LoRA: Learning Low-Rank Weight Updates
Low-rank weight updates, adapter gradients, and parameter-efficient fine-tuning.
Coming soon
LLM Training & Efficiency
QLoRA: Fine-Tuning Quantized Language Models
Four-bit weight quantization and low-rank adapters for memory-efficient fine-tuning.
Coming soon
Policy Distillation
Policy Distillation, Part i: Learning from an Expert Policy
Comparing action labels, Q-values, and probability targets when learning from an expert policy.
Coming soon
Policy Distillation
Policy Distillation, Part ii: Tokens and Complete Responses
Matching a teacher’s next-token probabilities or complete responses to train a language model.
Coming soon
Policy Distillation
Policy Distillation, Part iii: Learning on the Student’s Own Responses
Using teacher feedback on student-generated responses to address differences between training and generation.
Coming soon
Policy Distillation
Policy Distillation, Part iv: The MiniLLM Policy-Gradient Connection
Matching student and teacher response distributions through reverse KL divergence and policy-gradient updates.
Coming soon
Policy Distillation
Policy Distillation, Part v: Learning from Feedback with SDPO
Using feedback to build a self-teacher and guide token-level policy updates.
Coming soon
Language Model Architectures
FlashAttention: Computing Attention without Storing the Full Matrix
Blockwise attention and online softmax without storing the full attention matrix.
Coming soon
Language Model Architectures
Mixture of Experts: Choosing Which Parameters to Use
Sparse expert routing, load balancing, and the distinction between total and active parameters.
Coming soon
Language Model Architectures
Diffusion Language Models: Generating Text by Denoising
Forward masking, reverse denoising, and training objectives for non-autoregressive text generation.
Coming soon
LLM Training & Efficiency
Scaling Laws: Allocating Compute between Parameters and Data
Empirical loss scaling and compute-budget tradeoffs between model size and training tokens.