Knowledge distillation
Distillation done by matching the teacher’s full confidence spread, not just its final answers.
Think of it like
Like learning from a mentor who not only gives the answer but tells you how sure they were about each option — the near-misses teach you too.
Example
A student trains on the teacher’s soft probabilities (“80% cat, 15% dog, 5% fox”), absorbing nuance a hard “cat” label would hide.
How it actually works
Classic knowledge distillation, from Hinton and colleagues, trains the student on the teacher’s softened output distribution rather than one-hot labels. Those “soft targets” carry dark knowledge — the relative likelihoods across wrong answers — which conveys more than a single correct label. This lets small models learn faster and generalize better than training on hard labels alone.
For product teams
The technique behind small models that punch above their weight; the teacher’s uncertainty is part of the lesson.
For engineers
Train the student on the teacher’s soft (temperature-scaled) output distribution; soft targets carry extra signal.
Related
- Distillation — The core method within distillation.
- Student model — Produces a student model.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome