Sequence to Sequence

Read a whole input sequence, then produce a whole output sequence of a different length. Translation is the case that motivated it, and the framing was the contribution: once any task is a sequence in and a sequence out, summarising, answering and translating are the same problem with different data.

The original shape reads the input into a single fixed vector and generates from it. That vector is the entire connection between the two halves, which is elegant and is a bottleneck — everything about a fifty-word sentence has to survive in the same width as everything about a five-word one. Attention was introduced to remove exactly this, by letting the decoder look back at every input position rather than at one summary of them.

The bottleneck is a fixed-capacity channel carrying a payload that grows with the input, so the bits per input token fall as the input lengthens — which predicts, correctly, that quality degrades with source length and that the degradation is not gradual near the capacity. Attention makes the channel grow with the input instead, at the cost of a quadratic term, and every architecture since has taken that trade.