Attention Head
One independent attention computation inside a layer, specializing in a particular kind of relationship.
Think of it like
A single lane of traffic among several — each carries its own flow, all part of one highway.
Example
Researchers have found individual heads that track subject-verb agreement or copy the previous token.
How it actually works
A head is one slice of multi-head attention: it projects tokens into its own subspace and computes its own attention pattern. Different heads learn to specialize — some track position, some syntax, some coreference — and interpretability research often studies individual heads to understand what a model has learned. Modern efficiency tricks share keys and values across heads to save memory.
For product teams
The unit researchers poke at to understand what a model actually pays attention to.
For engineers
A single attention operation over one projected subspace within multi-head attention.
Related
- Multi-Head Attention — Many heads make multi-head attention.
- Attention — Each computes attention.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome