Reference

Glossary

39 terms, defined the way they are used in this course. Each links to the chapter that builds it.

All-reduceS17: A collective that sums a tensor across all ranks and gives every rank the result. Tensor parallelism needs one per sublayer, and it is blocking.
Arithmetic intensityS03: FLOPs performed per byte read from memory. An H100 needs roughly 300 to saturate; a decode step delivers about 1. The optimisations in this course either raise that number or route around it.
AWQS08: Activation-aware weight quantization. Identifies the small fraction of salient channels from activation statistics and scales them up before quantizing so they land on finer codes.
Block tableS06: A per-request array mapping logical KV block index to physical block index. It is a page table, applied to attention.
BubbleS17: Idle time in a pipeline while stages fill and drain. Fraction is (stages−1)/(micro-batches+stages−1).
Capacity factorS16: How many tokens an MoE expert may accept relative to a uniform share. Overflow tokens skip the expert silently.
Chunked prefillS11: Splitting a long prefill across several steps and mixing the chunks into decode batches, so no step is long enough for a streaming user to notice.
Continuous batchingS09: Admitting and retiring requests between forward passes rather than between batches. Also called iteration-level or in-flight batching.
cu_seqlensS09: The cumulative-sequence-lengths array that describes a ragged batch: a flattened token buffer plus offsets, with no padding.
DecodeS01: The phase after prefill: one token per forward pass, memory-bandwidth-bound, thousands of steps per request.
DisaggregationS18: Running prefill and decode on separate machines and shipping the KV cache between them, so their SLOs stop competing.
Effective bitsS08: Bits per weight including the per-group scales. A '4-bit' format is really 4.25–4.8, depending on group size and whether it stores a zero-point.
FlashAttentionS12: Tiled attention using an online softmax so the N×N score matrix is never written to HBM. The result is exact rather than an approximation.
GQAS03: Grouped-query attention: one KV head shared by several query heads. Shrinks the KV cache proportionally at almost no quality cost.
Head-of-line blockingS10: Under strict FCFS, one very large request at the front of the queue delays everything behind it.
Internal fragmentationS06: Memory reserved but never written. With paging, expected waste is block_size/2 tokens per request (worst case one partly-filled block), regardless of length.
ITL / TPOTS19: Inter-token latency, or time per output token: the gap between successive streamed tokens. Report it as percentiles rather than a mean.
Jump-forward decodingS15: When a grammar permits exactly one continuation, emit it without a sampling decision. A real engine still runs the model over the forced span to fill the KV cache, but as one batched pass instead of many, which is what makes constrained generation faster than unconstrained.
K-quantS08: GGUF's nested-scale format: 32-weight sub-blocks with quantized scales, grouped into 256-weight super-blocks with an fp16 scale-of-scales.
KV cacheS05: The stored keys and values of every processed token. Turns an O(N³) engine into an O(N²) one and becomes the scarce resource.
min-pS04: Keep tokens with probability at least m × p_max. The threshold scales with the model's own confidence, so at high temperature it degrades more gracefully than top-p.
MoES16: Mixture of experts: several FFNs, of which a router activates k per token. Compute drops; memory does not.
Online softmaxS12: Computing softmax incrementally with a running maximum and sum, rescaling accumulated values by exp(m_old − m_new) whenever the maximum moves.
PagedAttentionS06: Storing the KV cache in fixed-size blocks addressed through a block table, so allocation is on demand and blocks can be shared.
PreemptionS10: Reclaiming a running request's KV memory when the pool is exhausted, by either recomputing it later or swapping it to host memory.
PrefillS01: The first forward pass over the whole prompt. Compute-bound, parallel across positions, and where TTFT is spent.
Prefix cachingS07: Reusing KV blocks whose token contents (and position) have been computed before, matched by a chained hash.
Ragged batchS09: A batch whose members have different sequence lengths, flattened rather than padded.
Rejection samplingS14: Accept a draft token with probability min(1, p/q) and otherwise resample from norm(max(0, p−q)). Makes speculative decoding provably lossless.
RMSNormS03: LayerNorm without mean subtraction or bias: x / rms(x) × g.
RoPES03: Rotary position embedding: position injected by rotating pairs of dimensions in q and k. Two incompatible layouts exist (NeoX and interleaved).
SLOS18: A service level objective, typically a TTFT ceiling and an ITL ceiling. Almost every scheduling decision is a trade between the two.
SmoothQuantS08: Migrating activation outliers into the weights by scaling channel i down in the activation and up in the weight, leaving the product unchanged.
Speculative decodingS14: A cheap proposer suggests k tokens; the target model verifies all k in one pass and accepts the longest valid prefix.
SwiGLUS03: The gated FFN used by modern decoders: (silu(x@W1) * (x@W3)) @ W2. That is three matmuls per block, where a classic FFN has two.
Token budgetS10: max_num_batched_tokens: the cap on tokens processed per step, which bounds step time and therefore inter-token latency.
top-p / nucleusS04: Keep the smallest set of tokens whose cumulative probability reaches p, including the token that crosses the threshold.
TTFTS19: Time to first token, measured from arrival, queueing included. Excluding queue time hides the failure you most need to see.
WatermarkS10: KV blocks kept free so running requests can still grow. Admitting to 100% deadlocks.