Masked Attention
Attention with certain positions deliberately hidden, so the model can’t look where it shouldn’t.
Think of it like
Covering the answers below a quiz question so you can only use what’s above it.
Example
A text generator masks future tokens so, while predicting word five, it can’t peek at words six and beyond.
How it actually works
Masking sets certain attention scores to negative infinity before the softmax so those positions get zero weight. The most common use is causal masking in decoders — hiding future tokens to preserve the left-to-right generation setup — but masks also ignore padding or enforce custom visibility patterns. It’s a simple, powerful lever over what each token can see.
For product teams
The guardrail that keeps a generative model from cheating by seeing the answer it’s supposed to predict.
For engineers
Attention with selected scores set to −∞ pre-softmax to zero out disallowed positions.
Related
- Causal Attention — The causal case is causal attention.
- Attention mask — Implemented with an attention mask.
- Decoder — Used in the decoder.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome