Shared KV Memory: From Transformers to DeepSeek
Causal attention, KV caching, and the memory savings from sharing context across Transformer layers.
Blog
Coming soon
FlashAttention: Computing Attention without Storing the Full Matrix
Blockwise attention and online softmax without storing the full attention matrix.
Coming soon
Mixture of Experts: Choosing Which Parameters to Use
Sparse expert routing, load balancing, and the distinction between total and active parameters.
Coming soon
Diffusion Language Models: Generating Text by Denoising
Forward masking, reverse denoising, and training objectives for non-autoregressive text generation.