AdamW
Also called Adam with decoupled Weight decay
A fixed-up version of Adam that handles weight decay correctly — now the standard for training big models.
Think of it like
Like the same trusty cruise control after a recall fixed a subtle wiring bug — same idea, but it behaves the way it should.
Example
Most large language models are trained with AdamW because its weight decay generalizes better than plain Adam’s.
How it actually works
In original Adam, weight decay got entangled with the adaptive scaling, weakening its regularizing effect. AdamW decouples weight decay from the gradient update so it acts as intended, improving generalization. That subtle correction made it the default optimizer for transformer and large-model training, usually paired with a warmup-and-decay learning rate schedule.
For product teams
The optimizer behind most modern large models; a small fix that meaningfully helps generalization.
For engineers
Adam with decoupled weight decay; standard for transformer/LLM training, typically with warmup + decay.
Related
- Adam — Fixes Adam.
- Weight decay — The decay term is weight decay.
- Warmup — Paired with warmup.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome