Stochastic gradient descent
Also called Stochastic Gradient Descent
Gradient descent that updates using small random batches instead of the whole dataset, for speed.
Think of it like
Like adjusting a recipe after each small tasting rather than cooking the entire banquet before your first tweak.
Example
Instead of processing all million examples per step, the model updates weights after each batch of 32, taking many more, faster steps.
How it actually works
SGD estimates the gradient from a random mini-batch, making each step cheap and frequent; the noise it adds can even help escape shallow bad minima. It converges far faster in wall-clock time than full-batch descent on large data, at the cost of a noisier path. It’s the practical backbone of deep learning, and modern optimizers like Adam are refinements of it.
For product teams
The reason training on huge datasets is even feasible — cheap, frequent updates instead of one giant one.
For engineers
Gradient descent on random mini-batches; cheaper, noisier steps that scale to large data and aid escaping minima.
Related
- Gradient descent — A scalable form of gradient descent.
- Batch size — Each update processes one batch size of examples.
- Adam — Refined by Adam.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome