Ensemble Methods

Combine several models instead of trusting one. If their mistakes are different, averaging cancels some of them, and the combination is better than any member — which sounds like something for nothing and is not: what you spend is the compute to train and run several models.

The families differ in how they make the members disagree. Bagging trains each on a different resample and averages, which reduces variance and is what a random forest does. Boosting trains each on what the previous ones got wrong and adds them, which reduces bias and is what gradient boosting does. Averaging independently-initialised deep networks is the bagging idea without the resampling, and it works because random initialisation supplies the disagreement on its own.

The variance of an average of m models with pairwise correlation ρ falls to ρ + (1−ρ)/m of a single model's — so the ceiling is set by the correlation and not by the count. Perfectly correlated members give no reduction however many you add, which is the whole reason the methods work so hard at making members differ, and why the third and fourth member buy so much less than the second.