Dropout

Dropout switches off a random subset of units on every training step, so no unit can rely on any particular other one being there. The network is forced to spread its representation around instead of building fragile chains, and what remains works better on data it has not seen.

Zero each activation with probability p during training and scale the survivors by 1/(1−p) so the expected value is unchanged; disable it entirely at inference. Large language models use very little or none of it — with enough data, memorisation is less of a threat than underfitting, and the noise costs more than it buys.

h ← h·m/(1−p) with m ~ Bernoulli(1−p) elementwise. The 1/(1−p) factor keeps E[h] fixed so train and test distributions match, which is why the alternative of scaling at test time is equivalent but less convenient.