S04The Model·206 lines

Sampling

The sampler is the cheapest part of the engine and the part users notice most. Temperature, top-k, top-p, min-p and penalties are all logit surgery.

temperature
top-k
top-p
min-p
repetition penalty
seeding

Motivation

The problem

The model hands you 130,000 numbers. You need one token. Taking the largest, which is greedy decoding, is defensible for code and arithmetic and terrible for prose: it loops, it produces the blandest possible phrasing, and it makes the model look worse than it is.

Sampling from the full softmax is the opposite failure. The tail of a 130k-token distribution contains tens of thousands of tokens with individually negligible probability whose combined mass is several percent. Sample often enough and you will hit one, and a single absurd token derails everything after it, because autoregression conditions on its own mistakes.

So every engine ships a pipeline of filters. The pipeline costs microseconds and determines almost everything users mean when they say a model "feels" good or bad.

Core idea

The solution

Truncate the tail before sampling. The four standard truncations, in the order they are applied:

Temperature — divide logits by T before the softmax. Below 1 sharpens, above 1 flattens. T → 0 is greedy.
top-k — keep the k highest-probability tokens. Simple, but a fixed k is wrong at both ends: too permissive when the model is confident, too restrictive when it is uncertain.
top-p (nucleus) — keep the smallest set whose cumulative probability reaches p. It adapts to the shape of the distribution, and it became the default for that reason.
min-p — keep tokens with probability at least m × p_max. Newer, and better behaved than top-p at high temperature: the threshold scales with the model's own confidence rather than with an absolute mass target.

the sampler, completepython

def sample(logits, temperature=1.0, top_k=0, top_p=1.0, min_p=0.0, rng=None):
    if temperature <= 0:
        return int(logits.argmax())        # greedy: every other knob is ignored

    probs = softmax(logits / temperature)
    probs = apply_top_k(probs, top_k)
    probs = apply_top_p(probs, top_p)
    probs = apply_min_p(probs, min_p)
    probs /= probs.sum()                   # renormalise over survivors

    return int(rng.choice(len(probs), p=probs))

Order is part of the specification

Penalties act on logits; truncations act on probabilities; renormalise last. Apply top-p before temperature and you have a different sampler with the same parameter names. That is why the same temperature=0.7, top_p=0.9 can produce noticeably different behaviour across two engines serving the same weights.

Mechanics

How it works

Diagramthe sampling pipeline

Every stage is a mask over the vocabulary. Once a token is zeroed no later stage can recover it: filters compose by intersection.

The three penalties, which are not the same thing

Repetition penalty (multiplicative) divides positive logits of already-seen tokens and multiplies negative ones. Applied to the whole context, it suppresses the and destroys your grammar.
Presence penalty (additive, flat) subtracts a constant from any token that has appeared at least once.
Frequency penalty (additive, scaled) subtracts an amount proportional to how often the token has appeared.

The additive pair is better behaved because it does not interact with logit sign. If you only implement one, implement frequency penalty over a bounded window: typically the last few hundred tokens rather than the entire context.

Batched sampling is where this gets interesting

A real engine samples for every request in the batch at once, and each request has different sampling parameters. That turns a scalar operation into a masked, per-row GPU kernel. vLLM keeps a SamplingMetadata structure holding per-request temperatures, penalties and seeds, and groups requests by which filters they need so the no-op path stays cheap.

It also means the naive implementation, sorting the vocabulary per row, costs a 130k-element sort per request per token. Production kernels avoid the full sort by finding the threshold with a radix-select or an iterative bisection on the probability value.

Explore

Try it

Simulatorsampler pipelinewatch which tokens get struck through

temperature1.00top-kofftop-poffmin-poffrepetition penalty1.00

tokens surviving: 20 / 20
entropy (bits): 2.85
sampled: dog
p(sampled): 9.9%

distribution — grey bar = before filtering, coloured = after

the*36.2%
a19.9%
cat*12.1%
dog9.9%
sat*6.0%
ran4.4%
jumped3.0%
quietly2.0%
onto1.6%
under1.3%
mat1.0%
log0.8%
sofa0.6%
roof0.4%
.0.3%
!0.2%
epistemic0.1%
borogove0.1%
qux0.0%
zzz0.0%

Tokens marked * already appeared in the output, so the repetition penalty divides their positive logits. Struck-through rows were zeroed by a filter and can never be sampled, no matter the seed.

400 draws with these settings

The empirical histogram is the distribution users actually experience. A filter does not just make outputs 'better'; it makes whole regions of the vocabulary unreachable. An over-tight top-k is how a model ends up looping.

Set temperature to 1.6 and compare top-p 0.9 against min-p 0.05. At high temperature the distribution flattens, so top-p's cumulative target reaches dozens of junk tokens while min-p's relative threshold excludes them. The whole argument for min-p is visible in a single comparison.

Build it

Implementation

code/s04_sampling.py (excerpt)python

def apply_top_p(probs, top_p):
    if top_p >= 1.0:
        return probs
    order = np.argsort(-probs)              # descending
    cumulative = np.cumsum(probs[order])
    # keep everything up to AND INCLUDING the token that crosses top_p
    cutoff = int(np.searchsorted(cumulative, top_p)) + 1
    keep = order[:cutoff]
    out = np.zeros_like(probs)
    out[keep] = probs[keep]
    return out


def apply_min_p(probs, min_p):
    if min_p <= 0.0:
        return probs
    threshold = min_p * probs.max()         # relative to the top token
    return np.where(probs >= threshold, probs, 0.0)

The off-by-one that silently breaks top-p

The token that crosses the threshold must be kept, not dropped. Drop it and top_p=0.9 on a distribution whose top token has probability 0.95 keeps zero tokens, and your renormalisation divides by zero. Every implementation has this + 1; now you know what it is for.

Run it locally

Implements the full pipeline, then runs 50,000 draws under a dozen parameter settings and prints the empirical entropy, the number of reachable tokens, and a chi-squared check that the sampler matches the filtered distribution it claims to implement.

$ python code/s04_sampling.py

Expect: A parameter sweep table, an assertion that seeded sampling is reproducible, and a demonstration that top-p and min-p diverge sharply as temperature rises.

Only NumPy is required — setup instructions.

Production notes

In production

vLLM — batched GPU sampler with per-request parameters, plus a "fast path" that skips the sort entirely when every request in the batch is greedy.
llama.cpp / picoLM — a chain of composable sampler structs the user configures in order, so the ordering question is explicit rather than implicit.
Seeding — reproducibility requires a per-request RNG stream, not a global one. With a global seed, the token you get depends on how many other requests happened to be in the batch, and bug reports become impossible to reproduce.

Exercises

1
Implement all three penalties. Construct a prompt where repetition penalty at 1.3 measurably damages grammar by suppressing function words, and you have the argument for windowing.
2
Replace the full argsort in top-p with a bisection search on the probability threshold. Verify identical output and measure the speedup on a 128k-entry vector.
3
Implement per-request seeded RNG streams and prove that a request's output is independent of what else is in the batch.

Continue

What's next

That completes a correct, slow engine: tokenize, forward, sample, loop. Every remaining chapter makes it faster, and the first of them is the largest single win available. S05 introduces the KV cache and turns an O(N³) engine into an O(N²) one.

Check yourself

Questions

Answer before you look. A wrong answer tells you more than a right one, because the explanation points at the part of the chapter to reread.

Check yourselfQuestion 1 of 6

Why does min-p degrade more gracefully than top-p as temperature rises?

Score 0/6