Learning rate schedule
A plan for changing the learning rate during training instead of holding it fixed.
Think of it like
Like driving: accelerate gently from a stop, cruise, then ease off as you approach your exit.
Example
Training warms up the rate for the first few hundred steps, holds it, then decays it smoothly toward zero near the end.
How it actually works
A schedule varies the learning rate over the course of training — common shapes include linear warmup followed by cosine or linear decay. Early warmup avoids destabilizing the fresh weights; later decay lets the model settle into a good minimum with smaller, finer steps. The right schedule can noticeably improve final quality and stability over any single fixed rate.
For product teams
A quiet lever on final model quality — how you ramp and cool the learning rate matters.
For engineers
Time-varying learning rate (e.g. warmup + cosine/linear decay); improves stability and final convergence.
Related
- Learning rate — Controls the learning rate.
- Warmup — Includes a warmup.
- Training — Used across training.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome