Learning rate
Also called Learning Rate
The size of the step training takes each time it adjusts the weights.
Think of it like
Like your stride walking downhill in fog — tiny steps are safe but slow; huge leaps are fast but you might overshoot the valley.
Example
Set the learning rate too high and the loss bounces around or explodes; too low and training crawls for days.
How it actually works
The learning rate scales how far each weight moves along the gradient per step, and it’s often the single most important hyperparameter. Too large and training diverges or oscillates; too small and it’s painfully slow or gets stuck. Because no single value is ideal throughout, most training uses a schedule that changes it over time, often with a warmup and decay.
For product teams
The dial most likely to make or break a training run — worth tuning before almost anything else.
For engineers
Scalar scaling the gradient step; the dominant hyperparameter, usually varied via a schedule with warmup/decay.
Related
- Gradient descent — Scales each gradient descent step.
- Learning rate schedule — Varied by a learning rate schedule.
- Warmup — Ramped by warmup.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome