Adam
Also called Adaptive Moment Estimation
A popular optimizer that gives each weight its own adaptive step size, so training just works more often.
Think of it like
Like cruise control that also adjusts for each hill and curve — it adapts the pace per situation instead of one fixed speed.
Example
A team reaches for Adam as the default optimizer because it trains reliably across many models with little tuning.
How it actually works
Adam combines momentum (a running average of past gradients) with per-parameter scaling based on recent gradient magnitudes, so parameters with small or noisy gradients still move sensibly. This adaptivity makes it robust and fast with minimal tuning, which is why it became a default. Its main quirk is how it handles weight decay, which the AdamW variant fixes.
For product teams
The dependable default optimizer — it trains most models well without fuss.
For engineers
Momentum + per-parameter adaptive step sizes from first/second gradient moments; robust default, superseded by AdamW for decay.
Related
- Optimizer — A kind of optimizer.
- Stochastic gradient descent — Improves on stochastic gradient descent.
- AdamW — Refined by AdamW.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome