Uncertainty / Methods
verifiedDeep Ensemble
Train the same architecture five times from different random seeds and keep all five. Where they agree, the answer is probably safe; where they disagree, you have found the model's own doubt. It is the least clever method in uncertainty quantification and, embarrassingly often, the one that wins.
The disagreement is real because random initialisation and data order land the runs in different basins, so they fail on different inputs. That independence is the whole product, and it is what averaging a single run's checkpoints does not give you. Under distribution shift it holds up better than any single-model calibration fix, which is the result that made ensembles the default baseline rather than a curiosity.
The cost is not subtle: with m members you pay m times the training and m times the inference, so the share of your compute spent on copies is (m−1)/m. It saturates fast — going from one model to two costs as much as everything after five put together, and the accuracy gain runs the other way, most of it arriving by the third member. That asymmetry is the entire design question, and it is why five is the number people quote.
extra-members holds 50% of the budget; rest holds the remaining 50%.
Compute spent on the extra members, against the one model you would have trained anyway, in models. Drag the members up to watch copies take the whole budget — the second one costs more than everything after the fifth, while the accuracy arrives the other way round.
Reviewed by opendroid · 2026-08-18
- arXiv:1612.01474 — Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
- arXiv:1906.02530 — Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift