Gradient clipping
Capping how big a gradient step can be, so one wild update doesn’t wreck the model.
Think of it like
Like a governor on an engine that prevents it from redlining and blowing up on a sudden surge.
Example
A rare huge gradient would send the loss to infinity, but clipping caps its size and training keeps going smoothly.
How it actually works
Occasionally gradients explode — becoming enormous — and a single unclipped step can push weights into garbage, spiking the loss to NaN. Gradient clipping bounds the gradient’s norm (or value) before the update, keeping training stable through rough patches. It’s cheap insurance widely used in training deep and recurrent networks, especially with high learning rates or long sequences.
For product teams
A stability safeguard that keeps long, expensive training runs from suddenly falling apart.
For engineers
Bounds gradient norm/value before the update to prevent explosions; standard for stability in deep/recurrent training.
Related
- Gradient — Tames the gradient.
- Gradient descent — Stabilizes gradient descent.
- Learning rate — Guards the learning rate.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome