Optimizer
The algorithm that decides exactly how to update weights from the gradients each step.
Think of it like
Like a hiking guide who takes the “downhill” reading and decides the actual footing — pace, momentum, and where to place each step.
Example
You switch the optimizer from plain SGD to AdamW and the model trains faster and more stably on the same data.
How it actually works
The optimizer turns raw gradients into weight updates, often adding momentum and per-parameter adaptive step sizes to speed and stabilize training. Choices range from simple SGD with momentum to adaptive methods like Adam and AdamW, each with its own hyperparameters. It’s a major determinant of training speed and final quality, and it works hand in hand with the learning rate schedule.
For product teams
A behind-the-scenes choice that shapes how fast and reliably a model trains.
For engineers
Maps gradients to parameter updates (momentum, adaptive rates); e.g. SGD, Adam, AdamW; paired with an LR schedule.
Related
- Gradient — Consumes the gradient.
- Stochastic gradient descent — A simple one is stochastic gradient descent.
- AdamW — A popular one is AdamW.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome