Decoder-Only Transformer

The original transformer had an encoder that read and a decoder that wrote. Modern language models keep only the decoder: one stack, causally masked, trained to predict the next token. Everything else — translation, answering, summarising — is expressed as a continuation of text rather than a separate mechanism.

A stack of causally masked blocks over a single sequence, with a language-modelling head on top. Dropping the encoder removes cross-attention and halves the architecture, and the masking means every position in a training sequence contributes a prediction. Task specificity moves out of the architecture and into the prompt.

p(x) = Π sub _t  p(x sub _t  | x sub _<t ), with each conditional produced by the stack at position t. The causal mask makes all n conditionals computable in one forward pass, so a sequence of length n yields n supervised examples for the cost of one.