Elo Rating

Rank things by who beats whom rather than by scoring each alone. Two models answer the same prompt, a person picks the better one, and each result nudges both ratings — the winner up, the loser down, by an amount that depends on how surprising the result was. It came from chess and it is now how frontier models are compared.

It fits this problem well because pairwise preference is much easier for a person to give than an absolute score, and because it needs no fixed test set to be contaminated. What it inherits is chess's assumptions: that skill is one number, and that it does not depend on the opponent. Neither holds for models — one may be better at code and worse at prose, and a single rating averages that away, so two models with the same Elo can be reliably different on any particular task.

The expected score is a logistic function of the rating difference, so a 400-point gap means the stronger player is expected to score about 91%, and the update moves a rating in proportion to how far the result was from that expectation. The consequence worth knowing is that the confidence interval shrinks with the square root of games played, so early ratings on a new model are noisy in a way the leaderboard's ordering does not show — and a rank based on a few hundred votes is a much weaker claim than the same rank on a hundred thousand.