Self-distillation
Also called Self-Distillation
A model teaching itself by training on its own best outputs, rather than learning from a bigger teacher.
Think of it like
Reviewing your own strongest essays and studying them to write even better ones next time.
Example
A model generates answers, keeps the ones a verifier approves, and fine-tunes on them — bootstrapping itself upward without a larger model.
How it actually works
Ordinary distillation transfers knowledge from a large teacher to a smaller student. Self-distillation drops the separate teacher: the model produces outputs, filters them for quality, and trains on the survivors, often over several rounds. It works because generating a good answer can be easier than learning to produce it consistently. Gains taper and can amplify the model's own biases, so filtering quality is everything.
For product teams
A way to improve a model without access to a bigger one — but it can entrench existing quirks.
For engineers
Train a model on its own filtered generations, iteratively, without a separate teacher network.
Related
- Distillation — The teacher-student version it is derived from.
- Synthetic data — The self-generated data it relies on.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome