Inference / Decoding
verifiedNucleus Sampling
Top-k always keeps the same number of tokens, whatever the model thinks. Nucleus sampling keeps the same amount of probability instead: take tokens in order until their total reaches p, then sample from those. Where the model is confident that is one or two tokens; where it is unsure it may be hundreds.
Also called top-p, and the default for open-ended generation. It adapts the cutoff to the distribution rather than fixing it, which is the whole argument against top-k. Typically p ≈ 0.9 to 0.95, and it composes with temperature — though applying both hard makes the interaction difficult to reason about.
Take the smallest set V sub p with Σ sub x∈V sub p p(x) ≥ p, then renormalise over it. |V sub p | varies per step by construction, which is the property being bought. Setting p = 1 recovers pure sampling; p → 0 recovers greedy decoding, so the parameter sweeps the whole range between them.
8 values. The left group decays steeply; the right group is 99% of the way to flat, and reads flatter than the left.
The distribution with the nucleus highlighted against the tail it excludes. Drag p down to watch the tail get cut away and the kept set close to the few tokens the model is confident about.
Reviewed by opendroid · 2026-08-04
- arXiv:1904.09751 — The Curious Case of Neural Text Degeneration