Causal Masking

A model trained to predict the next word must not be allowed to see it. Causal masking blocks every position from attending to anything later than itself, so the prediction at each step is made from the past alone. It is what makes a single forward pass over a sentence into n training examples at once rather than one.

Add negative infinity to the upper triangle of the score matrix before the softmax, so those weights come out as exactly zero. Masking after the softmax would be wrong: the denominator would still include the future. This is the difference between a decoder and an encoder, and it is why decoder attention patterns are always triangular.

Scores S sub _ij  + M sub _ij  where M sub _ij  = 0 for j ≤ i and −∞ otherwise, then row-wise softmax. exp(−∞) = 0 makes the masked weights vanish while the remaining row still sums to one. The lower-triangular structure is what allows the KV cache: position i's keys and values never change when position i+1 arrives.