Top-k Sampling

Sampling from the full distribution occasionally picks something absurd, because the tail holds thousands of tokens that are each unlikely but collectively probable. Top-k keeps only the k most likely tokens, renormalises, and samples from those. The tail is cut off rather than merely made improbable.

Sort the logits, keep the top k, set the rest to negative infinity, softmax, sample. One parameter, no tuning per prompt, and it composes with temperature. Its weakness is that k is fixed while the shape of the distribution is not — the same k is far too permissive where the model is confident and far too strict where it is genuinely uncertain.

Let V sub _k  be the k highest-probability tokens. Sample from p'(x) = p(x)/Σ sub _{y∈V sub k}   p(y) for x ∈ V sub _k  and 0 otherwise. The truncated mass Σ sub _{y∈V sub k}   p(y) varies enormously across steps, which is exactly the quantity nucleus sampling fixes instead of k.