State Space Model

Instead of attending over everything you have seen, carry a fixed-size state forward and update it at each step. That is a recurrent network's idea, and it failed at long sequences for decades because the state either forgot or exploded. Structured state space models fixed the parametrisation rather than the idea, and reach thousands of steps where attention runs out of memory.

The trick is that a linear recurrence can be evaluated as a convolution, so training runs in parallel over the whole sequence while inference runs step by step at constant memory. The state matrix is initialised from HiPPO, which is what makes long memory the default rather than something the optimiser has to discover — an initialisation doing the work an architecture usually gets credit for.

x sub _t  = Āx sub _t−1  + B̄u sub _t , y sub _t  = Cx sub _t , with Ā, B̄ the discretised continuous system. Because the recurrence is linear, unrolling it gives y = K * u for a kernel K whose entries are CĀ super ^k  B̄ — computable by FFT in O(L log L) rather than by an O(L) sequential scan. Inference keeps the recurrent form, so cost per token does not grow with context.