Sampling Temperature

One dial between always taking the most likely next token and picking almost at random. Turn it down and the model repeats itself confidently; turn it up and it wanders. It is the cheapest knob in generation and the one most often turned without knowing what it actually does.

It divides the logits before the softmax, so it changes how sharp the distribution is and never which token is on top. That is the whole reason it cannot fix a ranking problem: if the right token is fourth, no temperature makes it first — it only changes how often the ones below the top get chosen. So it trades diversity against reliability and does nothing else, and reaching for it to improve quality is reaching for the wrong dial.

Dividing logits by T and exponentiating is the same as raising each probability to the power 1/T and renormalising. Since that map is monotonic, the ordering is preserved for every T > 0, while T approaching zero concentrates all mass on the argmax and large T goes uniform. The consequence is exact rather than approximate: temperature can change any probability and no ordering, which makes it a diversity control and never a quality control.