EMA
Also called Exponential Moving Average
A running average of the model’s weights that leans on recent steps but never forgets the past.
Think of it like
A rolling average of your daily spending — today counts most, but last week still nudges the number.
Example
During training, an EMA copy of the weights is kept alongside the live one, and the smoother EMA version is what gets shipped.
How it actually works
EMA maintains a shadow copy of the weights updated as a decaying blend: mostly the previous average, slightly the newest step. This damps the noise of individual updates and usually produces a more stable, higher-quality model than the raw final weights. It costs almost nothing but a second copy of the parameters.
For product teams
A standard, low-risk way to make the shipped model a bit more reliable than the raw training output.
For engineers
A decayed running mean of parameters maintained during training, typically used for the evaluation weights.
Related
- Weight Averaging — A time-weighted flavor of weight averaging.
- Checkpoint Averaging — Often paired with checkpoint averaging.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome