Cross-Entropy Loss

For a model that outputs a probability distribution, cross-entropy asks a simple question: how much probability did you put on the right answer? Put a lot and the loss is small; put almost none and it is enormous. It punishes confident mistakes far harder than uncertain ones.

Negative log likelihood of the correct class under the predicted distribution. Paired with softmax it has an unusually clean gradient — predicted minus target — which is why the pair is ubiquitous and why implementations fuse them rather than composing them. Perplexity, the number usually reported for language models, is just its exponential.

ℓ = −Σ sub _c  y sub _c  log p sub _c , which for a one-hot target reduces to −log p sub _correct . With p = softmax(z), ∂ℓ/∂z = p − y. Perplexity is exp(ℓ) averaged over tokens, so a loss of 2.0 is a perplexity of about 7.4.