Gradient accumulation
Summing gradients over several small batches before updating, to fake a bigger batch than memory allows.
Think of it like
Like filling a big order in several small trips because your cart is tiny — you still deliver the full load, just in pieces.
Example
Your GPU fits only 8 examples, but you accumulate over 32 mini-batches to update as if the batch size were 256.
How it actually works
When the desired batch size won’t fit in memory, gradient accumulation runs several forward-backward passes, adds up their gradients, and applies one optimizer step, simulating a large effective batch. This lets modest hardware train with the stability benefits of big batches, trading time for memory. The effective batch size is the per-step batch times the accumulation count.
For product teams
How you get big-batch training on small hardware — trading wall-clock time for memory you don’t have.
For engineers
Accumulate gradients over N micro-batches, then one optimizer step; effective batch = micro-batch × N.
Related
- Batch size — Simulates a larger batch size.
- Optimizer — Defers the optimizer step.
- Iteration — Spreads one step across iterations.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome