Weights / Foundations
verifiedMerge vs Ensemble
Both take several models and combine them; the difference is when. An ensemble keeps all of them and combines their outputs at every query. A merge combines their weights once and keeps one model. The ensemble is more accurate and the merge is the one you can afford to serve.
Which to choose is usually decided by the serving constraint rather than by accuracy. An ensemble of five costs five times the inference forever; a merge of five costs the same as one, permanently. So merging wins wherever inference dominates the bill, which is most deployed systems, and ensembling survives where accuracy is worth a fivefold cost — offline scoring, competitions, and the reference number a paper reports.
The two are not approximations of each other. An ensemble averages functions and a merge averages parameters, and those coincide only where the model is linear in its weights over the region spanned. Fine-tunes of a shared base are close enough to that regime for the merge to retain most of the ensemble's gain; models from different initialisations are nowhere near it, and there the merge gets nothing while the ensemble still works.
ensemble-members holds 50% of the budget; rest holds the remaining 50%.
Inference an ensemble pays on every query, against the single forward pass a merge costs, in models. Drag the member count up to watch the ensemble's recurring cost take everything — the merge's bar never moves, which is the entire argument for it.
Reviewed by opendroid · 2026-08-18
- arXiv:2203.05482 — Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
- arXiv:2403.13187 — Evolutionary Optimization of Model Merging Recipes