Weights / Methods
verifiedModel Soup
Fine-tune the same base many times with different hyperparameters, then average the weights of the ones that came out well. Normally you would pick the best on a validation set and discard the rest; a soup keeps them, and the average usually beats the single best run.
Two recipes. The uniform soup averages everything, which is simple and can be dragged down by a bad run. The greedy soup sorts candidates by validation accuracy and adds each only if it improves the running average — slower, more robust, and the version people actually ship. Either way the result is one model, so serving costs nothing extra, which is the difference between this and keeping the runs as an ensemble.
The gain comes from variance reduction in weight space rather than in output space. Independent hyperparameter draws land at different points in the same basin, and averaging cancels the component of each that is idiosyncratic to its run while preserving what they agree on. It follows that the benefit falls as the runs become more similar — a soup of near-identical runs is one run — and that it needs the runs to share a basin, which is why they must share a base.
cancelled-variance holds 25% of the budget; rest holds the remaining 75%.
Run-to-run variance that averaging cancels, against the part every run shares and averaging cannot touch, in equal units. Drag the cancelled share up to watch it approach its ceiling — it stops where the runs stop differing, which is why a soup of near-identical runs is one run.
Reviewed by opendroid · 2026-08-18
- arXiv:2203.05482 — Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
- arXiv:1803.05407 — Averaging Weights Leads to Wider Optima and Better Generalization