Transformer Block

The transformer is one block repeated. Each block does two things: attention, where positions exchange information, and a feed-forward network, where each position thinks about what it received on its own. Both sit inside residual connections with normalization, and stacking the block dozens of times is the whole architecture.

Two sub-layers, each wrapped as x + F(LN(x)) in the pre-norm arrangement. Attention is the only place information crosses positions; the feed-forward layer is position-wise and holds most of the parameters. That split is what makes the block easy to parallelise and is the seam every sparsity method cuts along.

h = x + MHA(LN(x)); y = h + FFN(LN(h)). Parameters per block are roughly 4·d sub _model ² for attention projections and 2·d sub _model ·d sub _ff  for the feed-forward layer, so at the conventional d sub _ff  = 4·d sub _model  the block is about one third attention and two thirds feed-forward.