Model Soup
Average the weights of several fine-tuned models into one that’s often better than any single one.
Think of it like
Blending many cooks’ versions of the same soup — the average tastes more balanced than any one pot.
Example
You fine-tune the same model ten times with different settings, average all ten, and the soup beats the best individual run.
How it actually works
A model soup takes several models fine-tuned from the same base with different hyperparameters and simply averages their weights. Surprisingly, the average often generalizes better than picking the single best model, because averaging cancels out each run’s idiosyncratic errors. The catch: the ingredients must share a starting point or the soup curdles.
For product teams
Squeezes more accuracy out of training runs you already paid for instead of throwing the losers away.
For engineers
Uniform or greedy averaging of same-lineage fine-tuned weights to improve generalization.
Related
- Model Merging — A specific recipe for model merging.
- Weight Averaging — It’s weight averaging applied to fine-tunes.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome