Optimization
26 concepts, 26 reviewed and merged.
Generalization
- Compute-Optimal TrainingGiven a fixed compute budget, you can spend it on a bigger model or on more data.verified
- OverfittingA model that keeps improving on its training data while getting worse on everything else has stopped learning and…verified
- Scaling LawsLoss falls predictably as models, data and compute grow — smoothly, along a straight line on a log-log plot, over…verified
Objectives
Regimes
- Fine-TuningA pretrained model knows a great deal but will happily continue text rather than answer a question.verified
- LoRAFine-tuning every parameter of a large model means storing a whole new model per task.verified
- PretrainingPretraining is the long, expensive first phase where a model learns language itself by predicting the next token over…verified
- Prompt TuningInstead of changing a model to fit a task, prompt tuning changes what you put in front of it — but learns those…verified
- QLoRALoRA makes the trainable parameters small, but the frozen base model still has to sit in memory at full precision…verified
Regularization
- DropoutDropout switches off a random subset of units on every training step, so no unit can rely on any particular other one…verified
- RegularizationAnything you do to a model to stop it memorising the training set.verified
- Weight DecayA model with enormous weights has usually memorised something rather than learned it.verified
Schedules
Training
- AdamDifferent parameters need different step sizes, and nobody can tune them individually.verified
- AdamWAdamW is Adam with the weight decay applied separately from the gradient rather than folded into it.verified
- BackpropagationTo improve, a model needs to know how much each of its millions of parameters contributed to being wrong.verified
- Batch SizeBatch size is how many examples the model looks at before each update.verified
- Gradient ClippingOccasionally a batch produces an enormous gradient — a strange example, a numerical accident — and one full-size step…verified
- Gradient DescentThe gradient points in the direction the loss increases fastest, so stepping the other way makes it smaller.verified
- MomentumPlain gradient descent forgets everything between steps, so it zig-zags across narrow valleys instead of running…verified
- MuonMomentum-based optimizers take a step in whatever direction the gradient points, and for a weight matrix that…verified
- Stochastic Gradient DescentComputing the gradient over the whole dataset for every step is unaffordable.verified
- Straight-Through EstimatorA discrete step — rounding, picking an index, thresholding — has a derivative of zero almost everywhere, so gradients…verified