Feed-Forward Network

Attention decides which tokens a position should look at. The feed-forward network is what the position does with what it found — a small two-layer network applied to each position independently. Attention moves information between positions; the feed-forward block thinks about it in place.

Two linear projections with a non-linearity between them, applied identically at every position: up-project from d sub _model  to d sub _ff , activate, project back down. d sub _ff  is conventionally 4·d sub _model , which puts roughly two thirds of a transformer's parameters in a block that never looks sideways. Because it is position-wise, it is embarrassingly parallel and the natural seam at which to introduce sparsity.

FFN(x) = W₂ · σ(W₁x + b₁) + b₂ with W₁ ∈ ℝ super ^{d sub _ff ×d sub _model}   and W₂ ∈ ℝ super ^{d sub _model ×d sub _ff}  . σ was ReLU originally and is now usually GeLU or a gated variant such as SwiGLU, which splits the up-projection in two and multiplies one half through a sigmoid gate. Gated variants use d sub _ff  ≈ (8/3)·d sub _model  to hold the parameter count fixed against the extra projection.