Long-Range Dependency

Some sequences need information from thousands of steps back — a variable defined at the top of a file, a motif that recurs across a piece of music, a load pattern that repeats yearly. Architectures differ less in whether they can represent such a dependency than in whether gradient descent ever finds it.

Every family has a different failure. Recurrent networks vanish the gradient over long spans, convolutions need depth proportional to the log of the span, and attention represents it in one hop but pays quadratically to do so. Benchmarks that claim to measure this often do not: several long-context evaluations are solvable by retrieving one span, which is a different capability from integrating across many.

For a recurrence the gradient across k steps carries a factor of the Jacobian product, which vanishes or explodes geometrically unless the spectrum is controlled — the reason gating and orthogonal initialisation exist. Attention removes the factor entirely, at O(L²) cost, so the choice is a trade between how far information travels per layer and what a layer costs.