Checkpointing
Also called Gradient / Activation Checkpointing
Save memory during training by throwing away intermediate results and recomputing them later, trading extra compute for a smaller footprint.
Think of it like
Not keeping every rough draft on your desk — you rewrite a page when you need it instead of storing all of them.
Example
A model that would not fit on a single GPU suddenly trains fine once activation checkpointing is turned on, at the cost of a slightly slower step.
How it actually works
A normal backward pass needs the activations from the forward pass held in memory. Checkpointing keeps only a few of them and recomputes the rest on demand during backprop. You pay for it with a second partial forward pass, but peak memory drops sharply — often the difference between a run that fits and one that OOMs.
For product teams
One of the levers that lets a bigger model train on the hardware you already have.
For engineers
Discard intermediate activations and recompute them in the backward pass to cut peak memory at the cost of ~30% more compute.
Related
- Mixed precision — Often used together to squeeze a model onto the hardware.
- Training — A memory-saving trick during the training loop.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome