Multi-Head Attention
Also called MHA
Run attention several times in parallel, each “head” focusing on a different kind of relationship.
Think of it like
A panel of specialists reading the same document — one tracks grammar, one tracks tone, one tracks facts.
Example
One head links pronouns to nouns while another tracks verb tense, and their views get combined each layer.
How it actually works
A single attention pattern can only capture one type of relationship at a time. Multi-head attention splits the representation into several subspaces and runs attention independently in each, then concatenates the results. This lets the model attend to many kinds of structure — syntax, coreference, position — simultaneously, and it’s standard in every transformer.
For product teams
Why a model can juggle several kinds of context at once rather than fixating on a single pattern.
For engineers
Parallel attention over projected subspaces whose outputs are concatenated and mixed.
Related
- Attention — Parallel copies of attention.
- Attention Head — Each one is an attention head.
- Grouped-Query Attention — Slimmed down by grouped-query attention.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome