Minimum Description Length

The best model of some data is the one that lets you write the data down in the fewest bits — counting the model itself as part of the message. It turns Occam's razor into arithmetic, and it explains why a model with a billion parameters can still generalise: what matters is not how many parameters there are but how many bits it takes to specify them.

This is why weight decay, quantisation and low-rank adapters all look like the same move from a distance — each shortens the description of the model. It is also the cleanest account of why a language model that compresses text well is doing something more than memorising: the model's own size is in the bill, so a lookup table pays for itself and wins nothing.

Choose the hypothesis minimising L(H) + L(D|H) — the bits to describe the model plus the bits to describe the data given it. The second term is the negative log-likelihood, which is why maximum likelihood is MDL with a free model: drop L(H) and you have removed the only thing penalising complexity.