Foundations / Supervised
verifiedEarly Stopping
Watch performance on data the model is not training on, and stop when it stops improving. It is the simplest regularisation available and it costs nothing — no extra term, no hyperparameter to search, just a decision about when to quit — which is why it is in almost every training loop whether or not anyone calls it that.
The trap is what happens to the data you stopped on. Choosing the stopping point using a validation set makes that set part of the fitting procedure, so its score is no longer an unbiased estimate of anything — and it is optimistic by exactly the amount you selected for. A run that stops at the best of two hundred evaluated checkpoints has taken the maximum of two hundred noisy numbers, and reporting that maximum as performance is the error, not the stopping.
The bias is a maximum over evaluations, so it grows with how often you check rather than with how long you train — check every step and you select harder than checking every epoch, on the same run. That is why a third split exists: validation chooses the checkpoint, test estimates the performance, and a system that reports the validation number has reported the thing it optimised.
Loss over 2000 training steps, starting near 9.0. It falls to about 1.51, with 97% of the total improvement arriving in the first half. A second line shows held-out, ending higher at about 1.71.
A run scored on its training data and on data held back from it. Drag the holdout up to watch the second curve turn upward while the first keeps falling — the crossing is where stopping pays, and the number you read there is not an unbiased estimate of anything.
Reviewed by opendroid · 2026-08-18
- arXiv:1611.03530 — Understanding deep learning requires rethinking generalization
- arXiv:1710.09412 — mixup: Beyond Empirical Risk Minimization
Origin · not linkable
- Prechelt 1998 — Early Stopping — But When? · Neural Networks: Tricks of the Trade, LNCS 1524, Springer · doi:10.1007/3-540-49430-8_3