Recurrent Neural Network

Process a sequence one element at a time, carrying a hidden state forward. The state is the network's memory of everything it has read, compressed into a fixed-size vector — which is both the elegance and the limit, because everything from a thousand steps ago has to fit in the same vector as the thing you just read.

This was how sequences were modelled for a quarter of a century and the reason it lost is not accuracy. Each step depends on the one before, so the computation cannot be parallelised across the sequence during training — a transformer processes a thousand positions at once and a recurrent network processes them one after another. When compute became the binding constraint, an architecture that could not use it lost to one that could, whatever their per-parameter merits.

The fixed-size state is an information bottleneck that does not grow with the input, so the bits available to describe the past are constant while the past itself grows — every long sequence is compressed into the same budget, and what gets discarded is decided by the training rather than by the reader. Attention removed exactly this constraint by keeping every position addressable, at a cost quadratic in the length, which is the trade the two architectures represent.