Gradient Boosting

Train a weak model, look at what it got wrong, and train the next one to fix those mistakes. Repeat a few hundred times and add the results together. Each model is deliberately too simple to be any good alone, and the sequence is what does the work — which is why this still wins on tabular data that deep learning has had a decade to take.

Fit each new tree to the gradient of the loss with respect to the current prediction, which makes "the errors so far" precise rather than heuristic. Depth, shrinkage and tree count trade against one another, and regularisation matters more here than in most methods. It is sequential by construction, so it parallelises inside a tree and not across them.

F sub _m (x) = F sub _m−1 (x) + η·h sub _m (x), with h sub _m  fitted to the negative gradient −∂L/∂F evaluated at F sub _m−1  — gradient descent in function space rather than parameter space. The shrinkage η trades the number of trees against how much each is trusted, and small η with many trees is almost always better than the reverse.