Continued pretraining
Keeping the pretraining going on a new corpus so a model soaks up a fresh domain or language.
Think of it like
Like sending a well-read generalist to live in a new field for a year — same person, now fluent in the local dialect and specifics.
Example
A general model gets continued pretraining on a big pile of legal documents so it internalizes legal language before any fine-tuning.
How it actually works
Rather than task-specific fine-tuning, continued pretraining runs the same self-supervised objective on additional domain data to broadly shift the model’s knowledge. It’s useful for specialized fields, new languages, or fresher data. The main hazard is catastrophic forgetting — pushing too hard on new data can erode the general abilities the base model started with, so data mixing and careful rates matter.
For product teams
How you make a model natively fluent in your domain — deeper than fine-tuning, but riskier and pricier.
For engineers
Additional self-supervised pretraining on new-domain data; balances adaptation against catastrophic forgetting.
Related
- Pretraining — Extends pretraining.
- Catastrophic forgetting — Risks catastrophic forgetting.
- Fine-tuning — Broader than fine-tuning.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome