Grouped-Query Attention
Also called GQA
Let several attention heads share one set of keys and values to shrink memory during generation.
Think of it like
A few coworkers sharing one filing cabinet instead of each keeping a full duplicate set of folders.
Example
A model gives 32 query heads but only 8 shared key-value groups, cutting the memory its KV cache needs.
How it actually works
The KV cache — stored keys and values for every past token — dominates memory during long generations. Multi-query attention shares a single key-value set across all heads (fast, but can hurt quality); grouped-query attention is the compromise, letting groups of query heads share key-value sets. It keeps most of the quality while sharply reducing cache size and bandwidth.
For product teams
A quiet efficiency choice that makes long-context serving cheaper and faster.
For engineers
Attention where query heads are grouped to share key/value projections, shrinking KV-cache footprint.
Related
- Multi-Head Attention — A middle ground for multi-head attention.
- Multi-Query Attention — The extreme case is multi-query attention.
- KV Cache — Shrinks the KV cache.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome