Attention
The mechanism that lets a model decide which other words to focus on when interpreting each word.
Think of it like
Reading “it” in a sentence and instinctively glancing back to figure out what “it” refers to.
Example
To resolve “the trophy didn’t fit in the suitcase because it was too big,” attention links “it” to “trophy.”
How it actually works
Attention computes, for each token, a set of weights over all other tokens — how much each one matters right now — and blends their information accordingly. Those weights are learned and context-dependent, so the same word attends differently depending on its neighbors. It’s the idea that unlocked long-range understanding in language models.
For product teams
The breakthrough that let models track meaning across a whole passage instead of just nearby words.
For engineers
A weighted aggregation over token representations where weights come from query-key similarity.
Related
- Transformer — The engine of the transformer.
- Self-Attention — The version inside one sequence.
- Query, Key, Value — Built from queries, keys, and values.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome