Structure / Decoding
verifiedBeam Search Tradeoffs
Widening the beam finds sequences the model likes better. It also, past a small width, produces worse output — a result that looks like a bug and is not. If a better search yields a worse answer, then the thing the search is optimising was not what you wanted, and the search's own limitations had been hiding that.
Separate the two error sources and the confusion goes away. A SEARCH error is the decoder failing to find the sequence the model scores highest. A MODEL error is that sequence being bad. A narrow beam has plenty of the first, which masks the second; widening it removes search errors and exposes model errors that were there all along. Reported gains from beam width are therefore not gains in the model, and reported degradation at large widths is not a decoder bug.
The specific pathology is length. Sequence probability is a product of per-token probabilities, so every additional token multiplies by something below one and longer sequences score lower by construction. A wider beam explores more short candidates and finds the very short ones the model scores highest — which is why unnormalised beam search degenerates toward the empty string, and why length normalisation is not a tweak but a repair of the objective.
model-errors holds 11% of the budget; rest holds the remaining 89%.
Errors that remain because the model's highest-scoring sequence is bad, against errors from the search not finding it, in errors. Drag the beam width up to watch model error take everything — a wider beam does not fix the output, it stops hiding what was wrong with it.
Reviewed by opendroid · 2026-08-18
- arXiv:2010.02650 — If beam search is the answer, what was the question?
- arXiv:1808.10006 — Correcting Length Bias in Neural Machine Translation