Weight Averaging
Take the numerical average of a model’s weights across runs or checkpoints to get a steadier model.
Think of it like
Averaging several thermometer readings to smooth out the jitter in any single one.
Example
Instead of shipping the final training step, a team averages the last ten checkpoints and ships that smoother version.
How it actually works
Individual training snapshots bounce around the loss landscape; averaging their weights lands you in a flatter, more robust spot that tends to generalize better. It works across checkpoints of one run or across separate runs from the same base. The requirement is compatibility — you can’t meaningfully average models that started from different places.
For product teams
A cheap post-processing trick that often buys a small, free accuracy bump before shipping.
For engineers
Parameter-space averaging that lands in flatter minima with better generalization.
Related
- Model Soup — The mechanism behind model soups.
- EMA — A moving-average variant is EMA.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome