Causal Attention
Also called Autoregressive Masking
Attention where every token can see the past and present but never the future.
Think of it like
Writing a story where you can reread earlier pages but the next ones haven’t been written yet.
Example
In GPT-style generation, causal attention ensures token five is predicted from tokens one through four only.
How it actually works
Causal attention applies a triangular mask so position i attends only to positions ≤ i. This enforces the autoregressive assumption — predict the next token from prior context — and lets training on a full sequence teach every next-token prediction at once. It’s the reason decoder-only language models can be trained efficiently and generate coherently.
For product teams
The rule that makes left-to-right text generation consistent between training and real use.
For engineers
Triangular masking restricting each position to attend only to itself and earlier positions.
Related
- Masked Attention — A specific masked attention.
- Autoregressive — Enables autoregressive generation.
- Decoder — Defines the decoder.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome