Teacher forcing
Also called Teacher Forcing
During training, feed the model the real previous word instead of its own guess, so mistakes do not snowball.
Think of it like
A music teacher who plays the correct next note for a student mid-piece, so one slip does not derail the whole song.
Example
While training a language model, each step is conditioned on the true prior tokens from the text, not on what the model itself predicted.
How it actually works
Autoregressive models generate one token from the previous ones. Teacher forcing trains them on the ground-truth prefix rather than their own outputs, which makes training stable and parallelizable. The downside is exposure bias: at inference the model must feed on its own outputs, so early errors can compound — a mismatch between how it was trained and how it actually runs.
For product teams
A training shortcut that speeds learning but partly explains why long generations can drift.
For engineers
Condition each training step on ground-truth prior tokens; efficient but introduces exposure bias at inference.
Related
- Autoregressive — The generation style it trains.
- Next-token prediction — The objective it is used with.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome