Self-Attention
Attention a sequence pays to itself — each word looking at the other words in the same sentence.
Think of it like
Everyone at a meeting table turning to hear whoever’s most relevant to the current point.
Example
In one sentence, self-attention lets “bank” attend to “river” nearby and settle on the water meaning, not money.
How it actually works
In self-attention, the queries, keys, and values all come from the same sequence, so every token gathers context from every other token in that same input. Stacking many self-attention layers lets meaning propagate and combine across the whole passage. It’s the workhorse operation inside every transformer block.
For product teams
What lets a model understand a word in the context of the whole message around it.
For engineers
Attention where queries, keys, and values derive from one sequence, mixing information within it.
Related
- Attention — A specific use of attention.
- Cross-Attention — Contrast with cross-attention.
- Transformer — The core of the transformer.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome