Context
How inference got fast
Nothing in this course was invented in one place. Each technique answered a specific failure of the one before it, and remembering which failure is the easiest way to keep them straight.
The transformer. Encoder-decoder, sinusoidal positions, post-norm.
Everything after this is inference on some variant of this architecture.
Column/row tensor sharding with one all-reduce per sublayer.
Still the sharding pattern every engine uses. Written for training; inference inherited it unchanged.
One KV head shared by all query heads.
The first acknowledgement that the binding constraint is the KV cache rather than the weights. GQA is the compromise version.
Tiled attention with online softmax; the N×N score matrix never reaches HBM.
Reframed attention from a FLOPs problem into an IO problem, and the result is exact rather than approximate.
Iteration-level scheduling — admit and retire requests between forward passes.
Continuous batching. Every serving engine since is a descendant.
Second-order weight quantization; migrating activation outliers into weights.
Made 4-bit weights and INT8 activations usable, and with them large models fitted onto small GPUs.
Virtual memory for the KV cache: fixed-size blocks and a per-request block table.
Recovered the 60–80% of KV memory lost to fragmentation, and made prefix sharing possible. The single most influential systems paper in the field.
A cheap draft proposes k tokens; the target verifies all k in one pass, with rejection sampling.
Broke the one-token-per-forward-pass assumption without changing the output distribution.
Activation-aware weight quantization; nested group scales in GGUF.
Local inference became practical. picoLM and quant.cpp are direct descendants.
Slice a long prefill into chunks and mix them into decode batches.
Removed the latency spike a long prompt inflicts on every other user, and the starvation of prompts larger than the budget.
A radix tree over token sequences that is simultaneously the allocator and the prefix cache.
Token-granular prefix matching, plus jump-forward decoding for structured output.
Compile a schema or grammar to a token-level automaton and mask the logits.
Valid JSON stopped being a prompting problem and became a decoding guarantee.
Extra heads on the target model predicting several future positions from its own hidden state.
Made target-assisted speculation cheap and accurate enough to become a first-class engine option.
Run prefill and decode on separate machines and ship the KV cache between them.
Decoupled the two SLOs that had always been in tension on shared hardware.
Sparse mixture-of-experts at serving scale.
Broke the link between parameter count and per-token compute, while leaving the memory bill intact.
A rewritten engine core; a shared kernel library combining paging, GQA and flash attention.
Consolidation. The techniques stopped being papers and became defaults.
A pooled KV cache any decode instance can read, merging disaggregation with global prefix caching.
The architecture is now organised around the cache instead of the model.
picoLM (~2,944 lines of C), quant.cpp (single-header, KV compression), baseRT (Rust + Metal, one CLI). All three are young and small; read them for clarity rather than for production maturity.
The ideas above, re-implemented small enough to read in an afternoon. Reading one is how you find out whether you understood them.
Dates are when the idea entered general use in serving, not necessarily first publication. Several had been known elsewhere for decades: paging is from the 1960s, rejection sampling from 1951, online softmax from 2018. Most of the work was noticing they applied.