Instruction Tuning

A pretrained model continues text. Asked a question it may produce more questions, because that is what the training data often did next. Instruction tuning trains on pairs of instruction and good response until the model treats a prompt as something to answer rather than something to extend. The knowledge does not change; the default behaviour does.

Supervised fine-tuning on curated instruction-response pairs, sometimes called SFT. Tiny compared with pretraining — thousands to millions of examples against trillions of tokens — and it generalises to instruction types absent from the set, which is the surprising part. It is the first stage of every post-training pipeline and usually the base a preference method starts from.

Maximise Σ log p sub _θ (y | x) over instruction-response pairs (x, y), the same cross-entropy objective as pretraining with a different distribution and a much lower learning rate. Nothing in the objective preserves anything outside that distribution, which is why the mixture matters more than the count.