Inference / Decoding
verifiedBeam Search
Greedy decoding takes the most likely next token every step and can walk into a corner, because a mediocre token now sometimes opens a much better continuation. Beam search keeps several candidate sequences alive at once and extends them all, discarding the weakest each step. It searches a little rather than committing immediately.
Maintain b partial sequences, extend each by every token, keep the b best by cumulative log probability. Cost and memory scale with the beam width. It dominates machine translation and summarisation, and is largely absent from open-ended generation — where higher-probability text turns out to be repetitive and dull rather than better.
Score a sequence by Σ sub t log p(y sub t | y sub <t ), usually divided by length to stop the search preferring short outputs. The objective is a proxy: exact search over this score produces worse text than approximate search does, which is the empirical result that made beam search interesting to explain rather than merely to use.
8 values. The left group decays steeply; the right group is 67% of the way to flat, and reads flatter than the left.
Cumulative scores of the surviving candidates against the ones pruned beside them. Drag the beam width to watch the search keep more of the field alive.
Reviewed by opendroid · 2026-08-04
- arXiv:1908.10090 — On NMT Search Errors and Model Errors: Cat Got Your Tongue?
- arXiv:2010.02650 — If beam search is the answer, what was the question?