Reference

Run it locally

Every chapter but the last ships a self-contained Python file: it runs in under two seconds, prints what it measured, and asserts the claims the chapter makes. The only dependency is NumPy. The final chapter is TypeScript and runs with Node.

Setup

terminalbash

git clone <this repository>
cd learn-llm-inference-from-scratch

python3 -m venv .venv && source .venv/bin/activate
pip install numpy

Python 3.10 or newer, with no GPU, no model download and no PyTorch. The early chapters use a word-level bigram model, which lets the engine mechanics be exercised end to end without weights. Where the architecture itself matters (S03), the code is the real architecture with randomly initialised weights.

Running a chapter

terminalbash

python code/s01_generation_loop.py
python code/s06_paged_attention.py
python code/s14_speculative_decoding.py

python code/run_all.py

Two files do more than print. s19_server.py --serve starts a real OpenAI-compatible streaming server on localhost:8000, using the standard library alone rather than FastAPI. s20_complete_engine.py assembles everything and runs a leave-one-out ablation on your machine.

the serverbash

python code/s19_server.py --serve &

curl -N http://127.0.0.1:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"hi"}],"max_tokens":8}'

curl http://127.0.0.1:8000/metrics

The files

Chapter	File	Lines
S01The Generation Loop	code/s01_generation_loop.py	137
S02Tokenization	code/s02_tokenizer.py	232
S03The Transformer Forward Pass	code/s03_transformer.py	251
S04Sampling	code/s04_sampling.py	206
S05The KV Cache	code/s05_kv_cache.py	205
S06PagedAttention	code/s06_paged_attention.py	283
S07Prefix Caching	code/s07_prefix_caching.py	263
S08Quantization	code/s08_quantization.py	252
S09Continuous Batching	code/s09_continuous_batching.py	244
S10The Scheduler	code/s10_scheduler.py	442
S11Chunked Prefill	code/s11_chunked_prefill.py	190
S12FlashAttention	code/s12_flash_attention.py	183
S13Kernel Fusion & CUDA Graphs	code/s13_cuda_graphs.py	185
S14Speculative Decoding	code/s14_speculative_decoding.py	188
S15Structured Output	code/s15_structured_output.py	338
S16Mixture of Experts	code/s16_moe.py	152
S17Tensor & Pipeline Parallelism	code/s17_parallelism.py	216
S18Disaggregated Prefill/Decode	code/s18_disaggregation.py	186
S19The Serving Layer	code/s19_server.py	363
S20The Complete Engine	code/s20_complete_engine.py	421
S21picoLM, in Python	code/s21_picolm.py	495
S22mini-picoLM, in TypeScript	code/src/lib/minipicolm/	4018

A few files import from earlier ones: s07 uses the block manager from s06, and s20 uses both. Run them from inside code/, or through run_all.py, which sets the working directory for you.

Going further with real weights

When you want the same code running a real model, the smallest useful step is Llama-3.2-1B in fp16 with transformers for weight loading only. Keep your own forward pass from S03, and check it against HuggingFace generate() at temperature 0. If the first twenty tokens match, your RoPE layout and your attention masking are both right, and those are the two things most likely to be wrong.