Machine Translation

The task that produced the encoder-decoder, attention, and eventually the transformer. It is also the task where evaluation is hardest to take seriously: there are many correct translations of a sentence, and the standard automatic metric compares against one or two of them by counting shared word sequences.

BLEU is known to be a poor proxy for quality and remains the reported number because it is cheap and comparable to a decade of prior work. Learned metrics correlate far better with human judgement and are not comparable across papers that use different versions of them. Both facts are true at once, which is why the field reports BLEU and argues about it rather than replacing it.

BLEU is a geometric mean of n-gram precisions with a brevity penalty, so it rewards matching the reference's phrasing rather than conveying its meaning — a correct translation using different words scores badly. Score differences below roughly one point are not meaningful, and papers routinely report differences smaller than that.