Gradient descent
The optimization method that repeatedly nudges weights downhill to shrink the loss.
Think of it like
Like walking downhill in fog by always stepping in the steepest downward direction until you reach the valley.
Example
Each training step, the model computes how wrong it is, then adjusts every weight a little in the direction that reduces the error.
How it actually works
Gradient descent iteratively updates parameters in the opposite direction of the loss gradient, scaled by a learning rate, to find a minimum. Plain (batch) gradient descent uses the whole dataset per step, which is accurate but slow; that impracticality at scale is why stochastic variants took over. It’s the fundamental optimization engine under nearly all neural network training.
For product teams
The core mechanic of “the model learned” — countless tiny downhill steps on an error surface.
For engineers
Iterative param update opposite the loss gradient, scaled by learning rate; full-batch form is the baseline for SGD.
Related
- Gradient — Follows the gradient.
- Stochastic gradient descent — The scalable version is stochastic gradient descent.
- Optimizer — Run by the optimizer.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome