Positional Encoding

Attention has no idea what order anything is in. Shuffle the input and the output shuffles with it, unchanged. Positional encoding puts the order back by adding a position-dependent signal to each token's representation, so that a word at the start and the same word at the end no longer look identical.

The original scheme adds fixed sinusoids of geometrically spaced frequencies, chosen so that a fixed offset is a linear function of the encoding — which in principle lets the model learn relative positions. Learned absolute embeddings are the simple alternative and do not extrapolate past the trained length. Everything since has been an attempt to encode relative position more directly.

PE(pos,2i) = sin(pos/10000 super ^2i/d ), PE(pos,2i+1) = cos(pos/10000 super ^2i/d ), added to the token embedding. Wavelengths form a geometric progression from 2π to 10000·2π, so low dimensions vary quickly with position and high dimensions slowly.