22 chapters · 9,450 lines of code · no GPU required
Build an LLMinference enginefrom scratch.
Anyone can call an inference API. Explaining why vLLM is 40× faster than a small PyTorch loop is harder, and that is the gap this course closes. You start from a plain generation loop and improve it one bottleneck at a time.
Every chapter comes with a mechanism-level diagram, a simulator you drive yourself, six questions with worked explanations, and a file you can run that checks what the chapter claims.
Chapter 1 · the whole engine
01
Start with the loop; then remove its bottlenecks.
An engine calls the model, samples a token, appends it, and repeats. The remaining chapters deal with what that short loop hides: repeated work, idle hardware, and memory that runs out.
02
Almost every optimisation is a memory optimisation.
Decode reads the entire model to produce one token, so the bottleneck is bytes moved rather than arithmetic. The KV cache, paging, prefix reuse and quantization all follow from that, and between them they take up most of the course.
03
Production engines become easier to read.
Each chapter maps its central idea onto vLLM, SGLang, llama.cpp, picoLM, or quant.cpp, so there is always a concrete file to open next.
Who this is for
Four ways to arrive here
Prerequisites: Python, and a rough idea of what a transformer is. You will not need calculus, CUDA, or a GPU.
You ship on top of an inference engine
Learn which controls matter, and why. The chapters on batching, scheduling and serving tie configuration choices to behaviour you can watch happen.
S09–S11 · scheduling→You are interviewing for inference or systems roles
Assemble precise answers to the questions that keep coming up: why decode is memory-bound, and what PagedAttention puts into a page.
S05–S06 · the cache→You read the vLLM source once and bounced off it
The chapter-by-chapter source references map the compact implementations here onto vLLM, SGLang, llama.cpp and the rest, so you know which file you are looking at.
Engine comparison→You learn by experimenting
The twenty-two simulators run real implementations, not scripted animations. Starve the KV pool and watch a request get preempted. The engine runs in your browser, so checking a prediction takes seconds.
S06 · try the simulator→
The curriculum
Five layers, twenty chapters, two capstones
Each layer builds on a constraint the layer before it introduced. On a first pass, read the chapters in order; out of sequence, the dependencies stop being visible.
The Model
S01–S04Turn weights into tokens: tokenizer, transformer forward pass, sampler. This is the loop that exists before any optimisation.
- S01137 lines
The Generation Loop
Autoregressive decoding
An inference engine is a while-loop that feeds a model's own output back into itself. Everything else in this course is an optimisation of that loop.
- S02232 lines
Tokenization
Bytes to token IDs
The model never sees text. A BPE tokenizer compresses bytes into a fixed vocabulary, and every latency number you quote is measured per token rather than per character.
- S03251 lines
The Transformer Forward Pass
RMSNorm, RoPE, GQA, SwiGLU
A modern decoder block is seven operations. Write them once in NumPy and every kernel optimisation later has a reference to check against.
- S04206 lines
Sampling
Turning logits into a token
The sampler is the cheapest part of the engine and the part users notice most. Temperature, top-k, top-p, min-p and penalties are all logit surgery.
…which leaves the loop correct and unusably slow. So:
Memory & KV Cache
S05–S08Almost every inference optimisation is a memory optimisation. Cache the KV, page it, share it, compress it.
- S05205 lines
The KV Cache
Why decode is memory-bound
Without a cache, generating token N costs O(N²). With one it costs O(N), and the bottleneck moves from compute to memory bandwidth.
- S06283 lines
PagedAttention
Virtual memory for the KV cache
Contiguous per-request KV buffers waste 60–80% of KV-cache memory to fragmentation. Paging the cache into fixed-size blocks recovers nearly all of it.
- S07263 lines
Prefix Caching
Never compute the same prompt twice
Shared system prompts, few-shot examples and multi-turn chats mean most prefill tokens have been computed before. Hash the blocks and reuse them.
- S08252 lines
Quantization
Fewer bits per weight and per KV entry
Decode speed is bytes-moved-per-token, so halving the bit width nearly halves latency. What is left to decide is where the error goes.
…which makes memory the scarce resource, not FLOPs. So:
Batching & Scheduling
S09–S12One request wastes a GPU. Continuous batching, the scheduler, preemption, and chunked prefill keep it saturated.
- S09244 lines
Continuous Batching
Iteration-level scheduling
Static batching makes every request wait for the slowest one. Admitting and retiring requests between forward passes raises throughput several-fold.
- S10442 lines
The Scheduler
Queues, budgets and preemption
At every step the scheduler decides which requests run. Fairness, latency SLOs and out-of-memory recovery all live here.
- S11190 lines
Chunked Prefill
Stop long prompts stalling decodes
A 32k-token prefill blocks every decode behind it for a second. Slice the prefill into chunks and mix them into decode batches.
- S12183 lines
FlashAttention
Online softmax, no N² matrix
Attention never needs to materialise the score matrix. Tiling plus a running max and sum turns an O(N²) memory cost into O(N).
…which saturates one GPU with many requests. So:
Decoding Acceleration
S13–S16Break the one-token-per-forward-pass rule, and constrain what the model is allowed to say.
- S13185 lines
Kernel Fusion & CUDA Graphs
Killing launch overhead
At small batch sizes the GPU is idle waiting for the CPU. Fusing operations and replaying a captured graph removes hundreds of launches per token.
- S14188 lines
Speculative Decoding
More than one token per forward pass
A cheap draft proposes k tokens; the big model verifies all k in one pass. Rejection sampling makes the output distribution provably identical.
- S15338 lines
Structured Output
Grammar-constrained decoding
Valid JSON is not a prompting problem. Compile the schema to a state machine and mask every logit the grammar forbids.
- S16152 lines
Mixture of Experts
Sparse activation, dense memory
An MoE layer activates 8 of 256 experts per token. Compute drops, memory does not, and routing becomes a load-balancing problem.
…which is as fast as one machine gets. So:
Distributed Serving
S17–S20Split the model across GPUs, split prefill from decode across machines, and put an API in front of it.
- S17216 lines
Tensor & Pipeline Parallelism
One model, many GPUs
Split each matmul column-wise then row-wise and one all-reduce per sublayer is all the communication you need. Pipeline parallelism trades latency for capacity.
- S18186 lines
Disaggregated Prefill/Decode
Two workloads, two machines
Prefill is compute-bound and decode is bandwidth-bound. Run them on one GPU and neither hits its SLO, so separate them and ship the KV cache across.
- S19363 lines
The Serving Layer
OpenAI-compatible streaming API
Users see TTFT and inter-token latency, not FLOPs. The API layer streams SSE deltas, handles cancellation, and exports the metrics you tune against.
- S20421 lines
The Complete Engine
Every piece, wired together
Nineteen chapters of parts assembled into one engine: scheduler, paged cache, prefix reuse, speculative decoding, streaming server. Then benchmarked.
The Capstone
S21–S22Take a real 2,944-line C engine and rebuild it in Python, module for module. It is the only chapter whose reference implementation is somebody else's shipping code.
- S21495 lines
picoLM, in Python
A real engine, module for module
picoLM runs a 1.1B model on a $10 board with 256 MB of RAM in 2,944 lines of C. Port it to NumPy one module at a time and you find every chapter of this course inside it, including the two it leaves out on purpose.
- S224018 lines
mini-picoLM, in TypeScript
The same engine, running in this tab
Rebuild the engine a third time: one ArrayBuffer, an arena, and eighteen steps to a browser that generates text with no server, no WASM and no dependencies. The hard part in JavaScript is the memory, not the maths.
In every chapter
A mechanism-level diagram
Follow the data as it moves, and see where computation, memory traffic and latency enter the system.
An interactive simulator
Push the inputs until an edge case shows up. Exhaust the KV pool, move a sampling threshold, and watch what the system does about it.
A quiz with explanations
Six graded questions per chapter, each with an explanation that points back to the relevant idea.
A file you can run
Self-contained and NumPy-only, apart from the TypeScript capstone, which runs with Node. It asserts its own claims and finishes in under two seconds. Nothing to download, no PyTorch, no GPU.
Begin with the generation loop.
Chapter 1 is nine lines of Python and one uncomfortable measurement. The rest of the course follows from that measurement.