Reinforcement Learning & Generative AI
Shared KV Memory: From Transformers to DeepSeek
Causal attention, KV caching, and the memory savings from sharing context across Transformer layers.
Blog
Reinforcement Learning & Generative AI
GRPO: Learning from Groups of Responses
Group-relative reward normalization, clipped token updates, and KL regularization, with baseline identities and DAPO extensions.
Reinforcement Learning & Generative AI
LLM Alignment: From PPO to DPO
Deriving KL-regularized alignment, PPO's rollout objective, and DPO's preference loss, with a population identification proof.
Foundations
Bayesian Optimization: Learning Where to Evaluate Next
Gaussian-process conditioning, expected improvement, and a finite-domain GP-UCB regret bound, with reproducible Python examples.
Reinforcement Learning & Generative AI
MARL, Part 4: Multi-Agent Policy Gradients
Joint policy gradients, deterministic actor–critic updates, and the sequential importance corrections behind HAPPO.
Reinforcement Learning & Generative AI
MARL, Part 3: Centralized Training, Decentralized Decisions
Local information constraints, centralized critics, and why an agent’s memory matters even when training observes the state.
Reinforcement Learning & Generative AI
MARL, Part 2: Cooperation and Credit Assignment
Shared rewards, counterfactual policy gradients, and additive or monotonic value decomposition for cooperative agents.
Reinforcement Learning & Generative AI
MARL, Part 1: Foundations and Markov Games
Joint policies, best responses, and Nash equilibria, with a minimax Bellman equation and contraction proof for zero-sum Markov games.
Reinforcement Learning & Generative AI
Policy Gradients in Reinforcement Learning
Deriving the policy gradient and comparing return-based, actor–critic, natural, trust-region, and deterministic methods.
Reinforcement Learning & Generative AI
Reinforcement Learning, Part ii: Partial Observability, Offline Data, and DTRs
What changes with hidden states or fixed datasets, and how history-based offline learning connects to DTRs.
Reinforcement Learning & Generative AI
Reinforcement Learning, Part i: Foundations and Learning Methods
MDP assumptions, Bellman equations, a value-iteration proof, and the main families of RL methods.
Research Notes
CLT under Local Dependence: When Observations Are Connected
A dependency-graph CLT, its Stein-method proof, and Berry–Esseen bounds under local dependence.
Research Notes
Direct Learning of DTRs, Part iii: Multiple Treatments and SDSS
Pairwise score surrogates, simultaneous optimization, and a multistage Fisher-consistency proof.
Research Notes
Direct Learning of DTRs, Part ii: Smooth Surrogates and DTRESLO
From one-stage outcome-weighted learning to a smooth two-stage objective, with a value-regret proof.
Research Notes
Direct Learning of DTRs, Part i: Doubly Robust Value Search
Augmented value estimation, a double-robustness proof, and joint search over treatment rules.
Research Notes
Offline Policy Learning from Logged Data
Q-learning and backward induction for treatment decisions using logged longitudinal data.
Research Notes
Surrogate Loss Functions and Fisher Consistency in Binary Classification
Surrogate risk minimization and Fisher consistency for binary classification.
Foundations
Convex Optimization III: A Reference to Convex Functions
Convexity properties and derivatives of common losses, norms, and penalty functions.
Foundations
Convex Optimization II: Duality and KKT Conditions
Lagrangian duality, KKT conditions, and worked examples of constrained convex optimization.
Foundations
Convex Optimization I: Convexity and Examples
First- and second-order criteria for convexity, with worked examples.
Foundations
Support Vector Machine
Maximum-margin classification, hinge loss, kernels, and primal-dual formulations.
Reflections
Purpose in Everyday Choices
How everyday choices and reflection help us understand what matters to us.
Foundations
Conditional Expectation under Squared-Error Loss
Conditional expectation as the minimum mean squared-error predictor, with proof and applications.
Foundations
Multinomial Logistic Regression
Softmax probabilities, likelihood derivations, and estimation for multiclass outcomes.
Foundations
Binary Logistic Regression
Modeling binary outcome probabilities through likelihood estimation and Bayesian inference.
Foundations
Bayesian Linear Regression
Posterior inference and prediction in linear regression with Gaussian and Laplace priors.
Foundations
Linear Regression
Least squares, likelihood estimation, and ridge, lasso, and elastic-net regularization.
Foundations
Vector and Matrix Derivatives
Gradients, Jacobians, and matrix derivatives, with dimension conventions and worked formulas.
Foundations
Linear Algebra Essentials
Matrix identities for inverses, determinants, and traces, with common matrix decompositions.
Reflections
The Modern Life Trap
How the pursuit of the next achievement can keep us postponing the life we want.