Knowledge Distillation

Train a small model to copy a large one's outputs rather than the original labels. The large model's full distribution over classes says more than a single correct answer does — that this dog is somewhat wolf-like and not at all a truck — and the small model learns faster from that than from the label alone.

The temperature on the teacher's softmax is the main knob: raise it and the small logits carry usable information, leave it at one and almost all the signal is in the argmax. What actually transfers is contested — some of the gain is the extra information and some is regularisation from a smoother target — and distilling from a teacher's intermediate features rather than its outputs is a different method with different failure modes.

Minimise a weighted sum of cross-entropy against the labels and KL divergence against the teacher's softened distribution: L = α·CE(y, σ(z sub _s )) + (1−α)·T²·KL(σ(z sub _t /T) ‖ σ(z sub _s /T)). The T² factor is not decorative — softening scales the gradients by 1/T², so without it the distillation term's weight would change every time the temperature moved.