Rotary Position Embedding

Rather than adding position to a token's representation, RoPE rotates it — by an angle proportional to where it sits. Two tokens dotted together then depend only on the angle between them, which is the distance separating them. Relative position falls out of the geometry instead of being learned.

Rotate each pair of dimensions of Q and K by an angle proportional to position before the dot product. It touches queries and keys only, never values, and adds no parameters. The relative property holds exactly, which is why it extrapolates further than absolute schemes and why context-extension tricks work by rescaling its base frequency.

Group the d sub _k  dimensions into pairs and rotate pair i at position m by angle mθ sub _i  with θ sub _i  = 10000 super ^−2i/d . Then ⟨R sub _m  q, R sub _n  k⟩ depends on m − n alone. Scaling θ's base extends the usable context by slowing the rotation.