LoRA

Fine-tuning every parameter of a large model means storing a whole new model per task. LoRA freezes the original and learns a small low-rank correction beside it, so a task costs megabytes instead of gigabytes and the base model is shared across all of them.

Represent the weight update as the product of two thin matrices and train only those. Optimizer state shrinks with the trainable count rather than the model size, which is the memory saving that matters. The adapters can be merged into the weights after training, so inference costs nothing extra.

W = W₀ + BA with B ∈ ℝ super ^d×r , A ∈ ℝ super ^r×k  and r ≪ min(d,k), giving r(d+k) trainable parameters against dk. At r = 8 on a 4096×4096 matrix that is 65k parameters instead of 16.8M.