Weight decay
Gently shrinking weights toward zero during training so the model stays simple and generalizes.
Think of it like
Like a tidy-desk rule that quietly clears clutter each day, keeping only what earns its place.
Example
Adding weight decay stops a model from letting a few weights grow huge to memorize training quirks.
How it actually works
Weight decay penalizes large weights, nudging all of them slightly toward zero each step, which discourages the model from fitting noise and improves generalization. It’s closely related to L2 regularization, though how it’s applied differs between plain Adam and AdamW. Set too high it underfits by over-simplifying; too low and it barely helps — it’s a standard, mild regularizer in almost every training recipe.
For product teams
A quiet guard against overfitting that’s in nearly every training recipe.
For engineers
Penalty shrinking weights toward zero (≈ L2); improves generalization, decoupled in AdamW.
Related
- Regularization — A form of regularization.
- Overfitting — Reduces overfitting.
- AdamW — Decoupled in AdamW.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome