Optimization / Training
verifiedGradient Clipping
Occasionally a batch produces an enormous gradient — a strange example, a numerical accident — and one full-size step in that direction can undo hours of training. Clipping caps the size of the update while keeping its direction, so a bad batch costs a small step rather than the run.
Rescale the whole gradient when its global norm exceeds a threshold, typically 1.0. Clipping per-parameter instead changes the direction, which is why the global norm form is standard. A run whose loss spikes and does not recover is usually a run without it.
g ← g·min(1, c/‖g‖) for threshold c and global norm ‖g‖ over all parameters. Direction is preserved exactly; only magnitude changes, and only when the threshold is exceeded.
8 values. The left group decays steeply; the right group is 51% of the way to flat, and reads flatter than the left.
Per-batch gradient norms before clipping and after. Drag the clipping up to watch the outliers pulled in while the rest pass through untouched.
Reviewed by opendroid · 2026-08-04
- arXiv:1211.5063 — On the difficulty of training Recurrent Neural Networks