Masked Language Model
Also called MLM
A training trick where you hide random words and make the model guess them from the words around them.
Think of it like
Like a fill-in-the-blank exercise — “The cat sat on the ___” — where you deduce the answer from the rest.
Example
BERT was trained by masking about 15% of words in each sentence and predicting each one.
How it actually works
By blanking out tokens and asking the model to recover them using both left and right context, MLM teaches deep bidirectional understanding without needing labeled data. It’s the training objective behind encoder models, but it doesn’t naturally produce a text generator. Next-token prediction won out for generative models precisely because it does.
For product teams
The learning recipe behind understanding-focused models, not chatbots.
For engineers
A self-supervised objective predicting masked tokens from bidirectional context.
Related
- BERT — The model type it trains.
- Next-token prediction — The generative training objective it contrasts with.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome