The five layers

Turn weights into tokens: tokenizer, transformer forward pass, sampler. This is the loop that exists before any optimisation.

1. S01The Generation LoopAutoregressive decodingAn inference engine is a while-loop that feeds a model's own output back into itself. Everything else in this course is an optimisation of that loop.137 ln
2. S02TokenizationBytes to token IDsThe model never sees text. A BPE tokenizer compresses bytes into a fixed vocabulary, and every latency number you quote is measured per token rather than per character.232 ln
3. S03The Transformer Forward PassRMSNorm, RoPE, GQA, SwiGLUA modern decoder block is seven operations. Write them once in NumPy and every kernel optimisation later has a reference to check against.251 ln
4. S04SamplingTurning logits into a tokenThe sampler is the cheapest part of the engine and the part users notice most. Temperature, top-k, top-p, min-p and penalties are all logit surgery.206 ln

You now have a correct engine that recomputes the entire past on every token.

Almost every inference optimisation is a memory optimisation. Cache the KV, page it, share it, compress it.

1. S05The KV CacheWhy decode is memory-boundWithout a cache, generating token N costs O(N²). With one it costs O(N), and the bottleneck moves from compute to memory bandwidth.205 ln
2. S06PagedAttentionVirtual memory for the KV cacheContiguous per-request KV buffers waste 60–80% of KV-cache memory to fragmentation. Paging the cache into fixed-size blocks recovers nearly all of it.283 ln
3. S07Prefix CachingNever compute the same prompt twiceShared system prompts, few-shot examples and multi-turn chats mean most prefill tokens have been computed before. Hash the blocks and reuse them.263 ln
4. S08QuantizationFewer bits per weight and per KV entryDecode speed is bytes-moved-per-token, so halving the bit width nearly halves latency. What is left to decide is where the error goes.252 ln

The engine is fast now, and the scarce resource is GPU memory rather than arithmetic.

One request wastes a GPU. Continuous batching, the scheduler, preemption, and chunked prefill keep it saturated.

1. S09Continuous BatchingIteration-level schedulingStatic batching makes every request wait for the slowest one. Admitting and retiring requests between forward passes raises throughput several-fold.244 ln
2. S10The SchedulerQueues, budgets and preemptionAt every step the scheduler decides which requests run. Fairness, latency SLOs and out-of-memory recovery all live here.442 ln
3. S11Chunked PrefillStop long prompts stalling decodesA 32k-token prefill blocks every decode behind it for a second. Slice the prefill into chunks and mix them into decode batches.190 ln
4. S12FlashAttentionOnline softmax, no N² matrixAttention never needs to materialise the score matrix. Tiling plus a running max and sum turns an O(N²) memory cost into O(N).183 ln

One GPU now stays saturated under many concurrent requests, and the kernels are what limits it.

Break the one-token-per-forward-pass rule, and constrain what the model is allowed to say.

1. S13Kernel Fusion & CUDA GraphsKilling launch overheadAt small batch sizes the GPU is idle waiting for the CPU. Fusing operations and replaying a captured graph removes hundreds of launches per token.185 ln
2. S14Speculative DecodingMore than one token per forward passA cheap draft proposes k tokens; the big model verifies all k in one pass. Rejection sampling makes the output distribution provably identical.188 ln
3. S15Structured OutputGrammar-constrained decodingValid JSON is not a prompting problem. Compile the schema to a state machine and mask every logit the grammar forbids.338 ln
4. S16Mixture of ExpertsSparse activation, dense memoryAn MoE layer activates 8 of 256 experts per token. Compute drops, memory does not, and routing becomes a load-balancing problem.152 ln

You have taken one machine as far as it goes. What remains does not fit on one.

Split the model across GPUs, split prefill from decode across machines, and put an API in front of it.

1. S17Tensor & Pipeline ParallelismOne model, many GPUsSplit each matmul column-wise then row-wise and one all-reduce per sublayer is all the communication you need. Pipeline parallelism trades latency for capacity.216 ln
2. S18Disaggregated Prefill/DecodeTwo workloads, two machinesPrefill is compute-bound and decode is bandwidth-bound. Run them on one GPU and neither hits its SLO, so separate them and ship the KV cache across.186 ln
3. S19The Serving LayerOpenAI-compatible streaming APIUsers see TTFT and inter-token latency, not FLOPs. The API layer streams SSE deltas, handles cancellation, and exports the metrics you tune against.363 ln
4. S20The Complete EngineEvery piece, wired togetherNineteen chapters of parts assembled into one engine: scheduler, paged cache, prefix reuse, speculative decoding, streaming server. Then benchmarked.421 ln

You now have an engine. The rest of the field refines these same steps.

Take a real 2,944-line C engine and rebuild it in Python, module for module. It is the only chapter whose reference implementation is somebody else's shipping code.

1. S21picoLM, in PythonA real engine, module for modulepicoLM runs a 1.1B model on a $10 board with 256 MB of RAM in 2,944 lines of C. Port it to NumPy one module at a time and you find every chapter of this course inside it, including the two it leaves out on purpose.495 ln
2. S22mini-picoLM, in TypeScriptThe same engine, running in this tabRebuild the engine a third time: one ArrayBuffer, an arena, and eighteen steps to a browser that generates text with no server, no WASM and no dependencies. The hard part in JavaScript is the memory, not the maths.4018 ln

You can now read a production engine end to end, because you have written one that agrees with it byte for byte where it is right, and disagrees where it is wrong.