Model checkpointing
Saving a snapshot of a model mid-training so you can resume or roll back to it later.
Think of it like
Like hitting “save” in a long video game so a crash doesn’t cost you hours of progress.
Example
A week-long training run writes a checkpoint every few hours; when a machine fails at hour 50, the team restarts from the latest snapshot.
How it actually works
Checkpointing periodically persists the weights (and optimizer state) so training runs survive crashes, can be evaluated at intervals, and can be resumed or reverted. It’s essential insurance for expensive runs, and it also lets teams pick the best-performing snapshot rather than whatever happened to be last.
For product teams
The reason a hardware failure during a costly run isn’t a catastrophe.
For engineers
Periodic persistence of weights and optimizer state enabling resume, evaluation, and rollback across a training run.
Related
- Checkpoint — A single saved snapshot.
- Training — The process it safeguards.
- Warm start — Resuming from a saved point.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome