Patch Embedding

Attention needs a sequence, and an image is a grid. Patch embedding cuts the image into fixed squares, flattens each one, and projects it to a vector — so a picture becomes a short sequence of tokens the same machinery can read. A 224-pixel image at 16-pixel patches is 196 tokens, which is an ordinary sentence length.

A single linear projection applied to each flattened patch, equivalent to a strided convolution with kernel and stride both equal to the patch size. Patch size is the whole trade: halving it quadruples the sequence, and attention costs the square of that — sixteen times the work for one halving — in exchange for finer detail. Position information has to be added separately, because flattening throws away the grid.

An H×W image at patch size P yields N = HW/P² tokens, each projected from ℝ super ^P²·C  to ℝ super ^{d sub _model}  . At 224×224 and P = 16 that is 196 tokens; at P = 8 it is 784, and attention cost rises as N² — sixteen times for one halving.