Entropy

How surprised should you expect to be? Entropy measures that in bits: a fair coin costs one bit per flip, a coin that always lands heads costs nothing, and a language model's entropy is how many bits it needs on average to say what comes next. Every compression bound and every training loss in this graph is a statement about it.

Entropy is the floor on lossless compression: no code can average fewer bits per symbol than the source's entropy, and arithmetic coding gets arbitrarily close. That is why a language model is a compressor — its cross-entropy on held-out text is a measured code length, in bits per token, and improving the model is literally shrinking the file.

H(X) = −Σ sub _x  p(x)·log sub ₂  p(x), in bits. It is maximal for the uniform distribution over n outcomes, at log sub ₂  n, and zero when one outcome is certain. Cross-entropy H(p,q) = −Σ sub _x  p(x)·log sub ₂  q(x) is what you actually pay for coding p's symbols with q's code, and it is never less than H(p) — the excess is exactly the KL divergence.