Quantization-Aware Training

Rather than quantise a finished model and hope, train with the rounding in the loop so the model learns weights that survive it. It costs a training run and reaches lower bit widths than post-training methods can — which matters below four bits, where simply rounding a trained model stops working.

Rounding has zero gradient almost everywhere, so training uses a straight-through estimator: quantise on the forward pass, pretend the operation was the identity on the backward one. That is an approximation nobody defends as exact, and it works, which is worth being uncomfortable about. The practical rule is that QAT is worth its cost only when the target precision is below what post-training quantisation already reaches.

Forward with ŵ = round(w/s)·s, backward with ∂ŵ/∂w ≈ 1 inside the representable range and 0 outside it. The estimator is biased, and the bias is what the model adapts to — training does not remove the rounding error, it finds weights where the error costs less.