Label Smoothing

Instead of training toward a target of exactly one for the correct class and zero for everything else, aim slightly lower — say 0.9 — and spread the remainder across the others. It is one line, it is in almost every transformer recipe, and it exists because the honest target is unreachable and chasing it does damage.

The unreachable part is literal: a softmax reaches one only in the limit of infinite logits, so a model trained toward one is trained to grow its logits without bound, which sharpens it and hurts calibration. Smoothing gives a finite optimum. The trade is that it deliberately destroys information — a smoothed model is worse as a teacher for distillation, because the between-class structure that distillation transfers is exactly what the smoothing erased.

With smoothing ε over K classes the target is 1−ε for the true class and ε/(K−1) elsewhere, and the optimal logit gap is finite rather than infinite. It also tightens the clusters a penultimate layer forms — representations of one class pull toward their centroid — which improves accuracy and calibration and removes the finer structure. Better predictions, worse representations, from one parameter: which of the two you need decides whether to use it.