Reference

References

What this course is built on. Read each paper after the chapter it belongs to; they get much easier once you have implemented the idea badly yourself.

Papers

Efficient Memory Management for LLM Serving with PagedAttentionKwon et al., SOSP 2023S06 S07
The vLLM paper. Introduces PagedAttention and measures the fragmentation it removes.
Orca: A Distributed Serving System for Transformer-Based Generative ModelsYu et al., OSDI 2022S09 S10
Iteration-level scheduling and selective batching: the origin of continuous batching.
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessDao et al., 2022S12
Tiling plus online softmax. The follow-ups (FA-2, FA-3) improve work partitioning and add FP8.
Online normalizer calculation for softmaxMilakov & Gimelshein, 2018S12
The recurrence FlashAttention is built on, published four years earlier for a different reason.
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-ServeAgrawal et al., OSDI 2024S11
Chunked prefill and stall-free batching.
Fast Inference from Transformers via Speculative DecodingLeviathan et al., 2022S14
The rejection-sampling proof that speculation preserves the target distribution exactly.
EAGLE / MedusaLi et al. 2024; Cai et al. 2024S14
Draft heads on the target model's own hidden states, and tree-shaped candidate verification.
GPTQ · AWQ · SmoothQuantFrantar et al. 2022; Lin et al. 2023; Xiao et al. 2022S08
The three families of post-training quantization: error compensation, salient channels, and outlier migration.
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head CheckpointsAinslie et al., 2023S03 S05
The compromise between multi-head and multi-query attention that every modern model now ships.
RoFormer: Enhanced Transformer with Rotary Position EmbeddingSu et al., 2021S03
RoPE. Read it if only to understand why the two implementation layouts are both called by the same name.
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismShoeybi et al., 2019S17
Column-then-row sharding with one all-reduce per sublayer. Written for training; inference uses it unchanged.
DistServe · Splitwise · MooncakeZhong et al. 2024; Patel et al. 2024; Qin et al. 2024S18
Prefill/decode disaggregation, and the KV-cache-centric architecture that follows from it.
Mixtral of Experts · DeepSeek-V3Jiang et al. 2024; DeepSeek-AI 2024S16
Sparse MoE at serving scale, including auxiliary-loss-free load balancing.

Engines worth reading

vllm-project/vllmreference implementationS06 S10 S14
Scheduler, KV cache manager, model runner, spec decode: most of this course in production form.
sgl-project/sglangRadixAttentionS07 S15
The prefix cache and the allocator as one radix tree; jump-forward decoding.
ggml-org/llama.cppGGUF and K-quantsS02 S08
The quantization formats the rest of the local ecosystem reads.
RightNow-AI/picolm2,944 lines of C11 at cf3f2dfS01 S05 S08 S12
mmap'd GGUF, fused dequant+dot, online softmax, grammar-constrained JSON, on a Raspberry Pi in 45 MB resident — the KV cache at a ~2k context, with the weights left file-backed.
quantumaikr/quant.cppKV cache compressionS08
Lloyd-Max codebooks, random Hadamard transforms, a full-precision window over recent tokens.
basecompute/baseRTlocal servingS19
One CLI: pull, chat, serve. Paged KV, continuous batching and prefix caching on Apple Silicon and GB10.
flashinfer-ai/flashinferkernelsS06 S12
Paged + GQA + flash attention in one library, used by several engines.
dottxt-ai/outlines · mlc-ai/xgrammarconstrained decodingS15
Schema and grammar compilation to token-level masks.

Write-ups

Inside vLLM: Anatomy of a High-Throughput LLM Inference SystemAleksa GordićS10 S20
The clearest end-to-end walkthrough of a production engine's internals.
vLLM blogthe vLLM teamS20
Release notes that double as design documents, the v1 engine post especially.
Transformer Inference ArithmetickipplyS05
Where the roofline numbers in this course come from, derived from first principles.

This course is an educational reimplementation and is not affiliated with any of the projects above. Where a chapter attributes a behaviour to a specific engine, check the current source: the field moves fast, and defaults change between releases.