Word Error Rate

The standard score for a transcript: line the hypothesis up against the reference, count the words you had to substitute, delete, or insert to repair it, and divide by the length of the reference. It is blunt on purpose. It does not know that "their" for "there" is worse than "a" for "the", and it will count a correctly spoken number written the other way as a miss.

Because the denominator is the reference length while insertions are unbounded, WER can exceed 100% — a model that hallucinates is not merely at chance, it is off the scale. Much of the disagreement between published numbers is text normalisation rather than acoustics: whether "%" is "percent", whether "Mr." matches "Mister". Whisper's authors reported real effort on a normaliser precisely because it moved results more than model changes did.

WER = (S + D + I)/N, with S, D and I the substitutions, deletions and insertions on the minimum-cost alignment and N the reference word count. That alignment is Levenshtein distance over words, computed in O(|ref|·|hyp|). Note the asymmetry: N counts the reference alone, so WER is not symmetric in its two arguments and is not one minus an accuracy.