Inference / Efficiency
verifiedQuantization
A trained weight does not need sixteen bits to be useful. Quantization stores weights in four or eight, which shrinks the model enough to fit on smaller hardware and — because decoding is bottlenecked on reading weights, not on arithmetic — usually makes it faster too. The loss in quality is far smaller than the loss in precision suggests.
Map each weight group to a low-bit integer with a scale, and dequantise on the fly. Post-training quantization needs no retraining and is what most deployments use. The difficulty is outliers: a few activation channels are orders of magnitude larger than the rest, and a naive scale spends its whole range on them and flattens everything else to zero.
For a group with scale s = max|w|/(2 super b−1 − 1), store round(w/s) and reconstruct as s·round(w/s). Error per weight is bounded by s/2, so it falls with the group size and rises with the group's largest magnitude — which is why outlier handling, not bit width alone, decides whether a scheme survives.
8 values. The left group decays steeply; the right group is 64% of the way to flat, and reads flatter than the left.
Weight magnitudes at full precision and after quantization. Drag the coarseness up to watch distinct magnitudes collapse onto the same level — which is what fewer bits buys.
Reviewed by opendroid · 2026-08-04
- arXiv:2208.07339 — LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
- arXiv:2210.17323 — GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers