S08Memory & KV Cache·252 lines

Quantization

Decode speed is bytes-moved-per-token, so halving the bit width nearly halves latency. What is left to decide is where the error goes.

INT8/INT4
group-wise scales
GGUF K-quants
AWQ/GPTQ
KV cache quantization

Motivation

The problem

S05 established that decode latency is bytes divided by bandwidth. An 8B model in fp16 is 16 GB; on a GPU with 3.3 TB/s you cannot generate a token faster than about 5 ms, however good your kernels are. On a laptop with 100 GB/s it is 160 ms, or six tokens per second.

There is exactly one way around a bandwidth floor: move fewer bytes. Quantization stores each weight in 4 or 8 bits instead of 16, which makes the model two to four times smaller and, if you do it right, two to four times faster.

The catch is that you are introducing error into every weight in the model on purpose. The craft is in deciding where that error goes.

Core idea

The solution

Affine quantization maps a range of floats onto a small set of integer codes:

quantize / dequantizepython

def quantize_group(w, bits=4):
    """One scale per GROUP of weights, not per tensor."""
    qmax = 2 ** (bits - 1) - 1
    scale = np.abs(w).max() / qmax        # symmetric: no zero-point
    q = np.clip(np.round(w / scale), -qmax - 1, qmax).astype(np.int8)
    return q, np.float16(scale)

def dequantize_group(q, scale):
    return q.astype(np.float32) * scale

The word doing all the work is group. Weights are the easy case; the truly extreme 10–100× outliers live in the activations, which are the harder problem taken up later in this chapter. Even so, per-channel ranges vary enough that quantizing a whole tensor with one scale wastes most of the code space. The scale stretches to fit the widest channel, every ordinary weight collapses onto two or three codes, and the model is destroyed.

Use one scale per 32 or 128 weights and an outlier only damages its own group. That single change is the difference between 4-bit working and 4-bit being useless.

"4-bit" is never 4 bits

Each group needs a scale, and sometimes a zero-point, stored in fp16. At group size 32 that is 16 bits of metadata per 32 weights, or half a bit each. Real 4-bit formats land at 4.5 bits/weight; GGUF names them honestly (Q4_K_M is ~4.8 bpw). When you compare formats, compare effective bits rather than the label.

Mechanics

How it works

Diagramgrouping, super-blocks, and fused dequantization

Left: why groups exist. Middle: how GGUF nests scales inside scales. Right: the rule that decides whether your quantized model is faster or slower than fp16.

The three families of weight quantization

Round-to-nearest (RTN) — quantize each weight independently. Free, requires no data, and good enough at 8 bits. At 4 bits it loses noticeable quality.
GPTQ — quantize column by column, and after each column update the remaining unquantized weights to compensate for the error just introduced. Uses second-order information from a small calibration set. Minimises the error in the layer's output, not in its weights.
AWQ — observes that a small fraction of channels matter disproportionately, identifies them from activation statistics, and scales them up before quantizing so they land on finer-grained codes. Cheaper than GPTQ and usually comparable.

All three reduce to the same structure, group-wise low-bit integers plus scales, even though the checkpoints they ship are different on-disk formats. Engines exploit that: convert each format into one internal layout and serve them all with one kernel. vLLM's Marlin path handles both GPTQ and AWQ this way.

Activation quantization is a different, harder problem

Weights are static and can be quantized offline with as much compute as you like. Activations are produced at runtime, vary with input, and carry far more extreme outliers, concentrated in a few channels.

SmoothQuant handles this by migrating difficulty from activations to weights: divide activation channel i by s_i and multiply the corresponding weight column by s_i. The product is unchanged, and both tensors are now quantizable. That matters because W8A8, with both quantized, uses the INT8 tensor cores; W4A16 only saves bandwidth.

bandwidth win only: W4A16; great for batch 1 decode; no gain at large batch
compute win too: W8A8; INT8 tensor cores — helps prefill and large batches
the current default: FP8; native on Hopper/Ada; near-lossless, no calibration

Quantizing the KV cache

At long context the KV cache is bigger than the weights, so it is the more valuable target. It is also harder: keys have strong per-channel outliers, and errors compound because a corrupted key affects every future token that attends to it.

The techniques that work in practice: quantize keys per-channel and values per-token (their outlier structures differ); keep a full-precision window of the most recent 64–128 tokens, since attention concentrates there; and apply a random Hadamard rotation before quantizing to spread outlier energy across all dimensions. quant.cpp combines all three and reports about 6× KV compression for roughly +0.1% perplexity.

Explore

Try it

Simulatorquantization error explorerset group size to per-tensor with outliers on

quantize

bits

group size32

effective bits/weight: 4.50
SNR: 13.2 dB
max abs error: 0.291
compression: 3.56×

128 weights · original vs reconstructed

original fp16 weight
reconstructed value
quantization error

Dashed lines are group boundaries; each group gets its own scale. Turn on outlier channels with group size set to per-tensor and watch the error explode everywhere: two extreme weights stretch the scale so far that all 126 ordinary weights collapse onto a handful of levels. Shrink the group and only the outlier's own group suffers. AWQ, GPTQ and SmoothQuant are all built on that one observation.

what it buys — 8B model, H100

fp16 baseline4.78 ms/token · 209 tok/s

4-bit, group size 321.25 ms/token · 799 tok/s

Decode latency is bytes ÷ bandwidth, so the speedup is almost exactly the compression ratio, provided the dequantization is fused into the matmul. Dequantize into a separate buffer first and you move more bytes than fp16 and end up slower.

memory footprint

weights: 4.2 GB
KV (unquantized): 16.0 GB
scale overhead: 11.1%
fits in 24 GB?: yes

Scales are stored in fp16 and are pure overhead. At group size 8 with 4-bit weights they add 2 bits per weight, a 50% tax. Group 32–128 is the sweet spot for that reason, and it is why '4-bit' models are really 4.5 bits.

weight seed11

The experiment that makes this chapter click: set bits = 4, group = per-tensor, and turn outliers on. SNR collapses. Now drag group size down to 32. The two outlier weights are still badly represented, but the other 126 recover completely. Two hundred lines of AWQ or GPTQ recover the rest.

Build it

Implementation

The kernel is where quantization succeeds or fails. The rule is absolute: dequantize inside the matmul, in registers, and never write float weights back to memory.

code/s08_quantization.py (excerpt)python

def fused_matmul(x, q, scales, group):
    """
    Dequantize inside the accumulation. Nothing full-precision is ever stored.
    q:      int8 codes, [n_groups, group, out_features]  — 4× smaller read
    scales: fp16,       [n_groups, out_features]
    """
    n_groups, _, out_f = q.shape
    out = np.zeros((x.shape[0], out_f), dtype=np.float64)

    for g in range(n_groups):
        sl = slice(g * group, (g + 1) * group)
        # dequantize this block in registers and accumulate immediately
        out += x[:, sl] @ (q[g].astype(np.float64) * scales[g].astype(np.float64))

    return out

If your quantized model is slower, this is why

The naive implementation calls dequantize() to build a full fp16 weight tensor, then calls the normal matmul. That reads the int4 weights and writes and re-reads an fp16 tensor, which is strictly more traffic than never quantizing at all. Every real engine fuses; picoLM calls its version fused dequant+dot and credits it with cutting memory traffic in half.

Run it locally

Quantizes a synthetic weight matrix at 2/3/4/8 bits across a range of group sizes, reports SNR, demonstrates the outlier catastrophe with per-tensor scales, and compares bytes moved for fused versus unfused dequantization.

$ python code/s08_quantization.py

Expect: A bits × group-size error table, a ~13 dB recovery from grouping on outlier weights, and a traffic table showing the unfused path moving more bytes than fp16.

Only NumPy is required — setup instructions.

Production notes

In production

GGUF K-quants (llama.cpp, picoLM, quant.cpp) — nested scales: 32-weight sub-blocks with 6-bit scales, grouped into 256-weight super-blocks with an fp16 scale-of-scales. This is why Q4_K_M beats plain Q4_0 at nearly the same size.
picoLM — dequantization kernels for Q2_K through Q8_0 with ARM NEON and x86 SSE2 paths, fused into the dot product.
quant.cpp — the KV-cache specialist: Lloyd-Max codebooks plus a random Hadamard transform, with a 128-token full-precision window.
vLLM / TensorRT-LLM — FP8 by default on Hopper, with AWQ and GPTQ INT4 paths for memory-constrained deployments.
baseRT — a proprietary .base format with affine Q2–Q8 and optional AWQ calibration, converted from GGUF/HF/MLX checkpoints.

Exercises

1
Implement per-channel (per-output-row) scales as well as per-group and measure which helps more for a fixed bit budget.
2
Implement the SmoothQuant migration: pick per-channel factors from activation statistics, fold them into the weights, and show the product is unchanged while both tensors quantize better.
3
Quantize the KV cache from S06 to int8 with per-token scales, then add a 128-token fp16 window and measure the perplexity difference on a fixed passage. Confirm the window matters more for keys than for values.

Continue

What's next

Memory is now as small and as shared as we can make it. The GPU is still mostly idle during decode, because one request cannot saturate it. S09 starts the batching layer, where throughput multiplies.

Check yourself

Questions

Answer before you look. A wrong answer tells you more than a right one, because the explanation points at the part of the chapter to reread.

Check yourselfQuestion 1 of 6

Why does per-tensor quantization fail catastrophically on layers with outlier channels?

Score 0/6