Distillation Dataset
A pile of a bigger model’s answers, used as the textbook to teach a smaller model to imitate it.
Think of it like
A recording of a master chef cooking, handed to an apprentice to study move for move.
Example
A team prompts a large model with 100k questions, saves its responses, and trains a small cheap model on that transcript.
How it actually works
Rather than distilling from a teacher’s internal probabilities, you often just collect its outputs — questions paired with the big model’s answers — and treat that as a training set for a smaller student. Quality depends entirely on the teacher and the prompts you chose. It’s cheap and effective, though the student inherits the teacher’s mistakes and blind spots too.
For product teams
How a small, cheap model gets much of a big model’s quality — by learning from its answers.
For engineers
A corpus of teacher-model outputs used as supervised targets to train a smaller student.
Related
- Knowledge distillation — The fuel for knowledge distillation.
- Teacher model — From a teacher to a student model.
- Synthetic data — A kind of synthetic data.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome