Attention mask
A yes/no map telling the model which tokens it’s allowed to look at and which to ignore.
Think of it like
Like blacking out lines on a page so a reader’s eyes skip them entirely.
Example
A padded batch uses an attention mask so the model treats real words as content and the trailing pad tokens as if they weren’t there.
How it actually works
The mask is a set of 1s and 0s applied inside attention: masked positions get their scores driven to negative infinity so they contribute nothing after softmax. Two big uses — hiding padding, and enforcing causality so a token can’t peek at future tokens in a decoder.
For product teams
A silent correctness layer; when it’s misaligned, models produce baffling garbage.
For engineers
Additive/boolean mask over attention logits setting disallowed positions to −inf pre-softmax; used for padding and causal masking.
Related
- Padding — The filler it usually hides.
- Where the masking is applied — inside attention.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome