Distillation
Compressing a big model’s know-how into a smaller one that mimics its behavior.
Think of it like
Like boiling a rich stock down to a concentrated cube — far smaller, most of the flavor kept.
Example
A large model’s answers train a compact model that runs ten times faster while behaving nearly the same on everyday tasks.
How it actually works
Distillation trains a small student to reproduce a large teacher’s outputs — its predictions or full probability distributions — packing more signal into fewer parameters than training from scratch. The payoff is speed, cost, and on-device deployment; the ceiling is the teacher, whose mistakes and biases the student inherits. It’s the standard bridge from a frontier model to a shippable small one.
For product teams
How you turn an expensive model into a cheap, fast one that’s good enough for most requests.
For engineers
Train a compact student to match a teacher’s outputs/soft targets; smaller and faster, capped by the teacher.
Related
- Knowledge distillation — The detailed form is knowledge distillation.
- Student model — Uses a student model.
- Teacher model — Guided by a teacher model.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome