Optimization / Regimes
verifiedQLoRA
LoRA makes the trainable parameters small, but the frozen base model still has to sit in memory at full precision, and that is what actually decides whether you can fine-tune on the hardware you have. QLoRA quantizes the frozen base to four bits and trains the adapters on top of it, which is what puts fine-tuning a large model on a single accelerator.
Freeze the base in 4-bit, keep the LoRA adapters in bf16, and dequantise each weight block only as it is used. Gradients flow through the quantized weights to the adapters, which stay full precision, so the loss in quality is much smaller than four bits suggests. Paged optimizer state handles the memory spikes that would otherwise abort a long run.
Memory goes from 2Ψ for a bf16 base to roughly 0.5Ψ at 4 bits, plus r(d+k) trainable parameters and their optimizer state. The base dominates either way, so the quantization rather than the adapter is what moves the total — LoRA shrinks the gradient side of the ledger and this shrinks the side that was already larger.
base-bits holds 80% of the budget; rest holds the remaining 20%.
Memory held by the frozen base against the trainable adapters beside it, in equal units. Drag the base precision and watch the base stay the majority at every setting — which is why quantizing it, not shrinking the adapter, is what moves the total.
Reviewed by opendroid · 2026-08-04
- arXiv:2305.14314 — QLoRA: Efficient Finetuning of Quantized LLMs