Multi-Query Attention
Also called MQA
All attention heads share a single set of keys and values — the most aggressive memory-saving variant.
Think of it like
The whole office sharing one filing cabinet — lean and fast, but everyone competes for the same drawers.
Example
An inference-optimized model uses one key-value head for all query heads to make its KV cache as small as possible.
How it actually works
Multi-query attention keeps many query heads but collapses to a single shared key and value head, minimizing the KV cache and the memory bandwidth needed each generation step. That makes decoding notably faster, at some cost to quality. Grouped-query attention emerged as the popular middle ground, recovering most of the lost quality while keeping much of the speed.
For product teams
A serving-speed optimization that trades a little quality for cheaper, faster generation.
For engineers
Attention with a single shared key/value head across all query heads, minimizing KV-cache size.
Related
- Grouped-Query Attention — The extreme of grouped-query attention.
- Multi-Head Attention — A variant of multi-head attention.
- KV Cache — Minimizes the KV cache.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome