Cross-Validation

One held-out test set gives you one number, and that number has error bars nobody prints. Cross-validation splits the data several ways, trains on each split, and averages — a steadier estimate of how the model will do on data it has not seen, bought by training it k times instead of once.

k-fold is the default and leave-one-out is the extreme. The failure that actually bites is leakage: any choice made using all the data — scaling, feature selection, hyperparameters — before the split makes every fold optimistic, and the resulting number is unearned. What it estimates is subtler than it looks, and is closer to the average error over datasets of that size than to the error of the one model you trained.

k-fold averages k estimates, each trained on (1 − 1/k) of the data, so it estimates the error of a model trained on slightly less data than the one shipped. The folds share training data and their estimates are therefore correlated, which is why the naive standard error across folds understates the real uncertainty.