Scaled Dot-Product Attention
The exact recipe for attention: score with dot products, shrink the scores, softmax, then blend.
Think of it like
Rating each option, dividing the scores to keep them sane, turning them into percentages, then mixing accordingly.
Example
Each query dots against all keys, the scores are divided by the square root of the dimension, softmaxed, and applied to values.
How it actually works
The “scaled” part matters: raw dot products grow with vector dimension, pushing softmax into saturated regions where gradients vanish. Dividing by the square root of the key dimension keeps scores in a workable range. After softmax turns them into weights, you take the weighted sum of values. This precise formula is the beating heart of the transformer.
For product teams
The standard, well-understood formula that every attention layer runs — rarely something you tune directly.
For engineers
Attention as softmax(QKᵀ/√d)·V, with √d scaling to stabilize gradients.
Related
- Attention — The concrete form of attention.
- Query, Key, Value — Operates on query, key, value.
- Softmax — Uses a softmax.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome