Warmup
Starting training with a small learning rate and ramping it up over the first steps.
Think of it like
Like stretching before a sprint — ease in so you don’t pull something at full speed from a cold start.
Example
For the first 500 steps the learning rate climbs from near zero to its target, so early updates don’t blow up the model.
How it actually works
At the start of training the weights are fresh and gradients can be large and erratic, so jumping straight to a high learning rate risks divergence. Warmup ramps the rate up gradually, letting the optimizer’s statistics and the weights stabilize before full-speed learning. It’s especially important for large models and adaptive optimizers, and is usually the front half of a full schedule.
For product teams
A small safeguard that prevents big training runs from destabilizing in their fragile first moments.
For engineers
Gradual learning-rate ramp over initial steps to stabilize early training; typically precedes decay.
Related
- Learning rate schedule — The opening phase of a learning rate schedule.
- Learning rate — Protects the learning rate.
- AdamW — Important for AdamW.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome