Position Interpolation

Rather than asking a model to handle positions it has never seen, squeeze the new positions into the range it already knows. A sequence four times too long is re-indexed so that position 16,000 is presented as position 4,000 — the model interpolates between distances it understands instead of extrapolating past them, which it cannot do.

The plain version scales every rotary frequency by the same factor, which works and costs resolution: neighbouring tokens now sit at distances the model reads as fractional, so fine-grained local ordering blurs. The refinements scale frequencies unevenly — high-frequency components, which carry local order, are left alone while low-frequency ones, which carry long-range position, are stretched. A short fine-tune on long sequences then recovers most of what the squeeze cost.

The trade is between the two things position has to encode. Compressing by a factor of s divides the angular separation between adjacent tokens by s, so local resolution falls in proportion — while the maximum representable distance rises by the same factor. There is no setting where both improve, which is why every method here is a choice about which frequency bands to spend the compression on rather than a way to avoid spending it.