KV Cache
Also called Key-Value Cache
Stored keys and values from earlier tokens so the model doesn’t recompute them every generation step.
Think of it like
Keeping your running notes open so you don’t reread the whole book each time you write a new line.
Example
While generating a long reply, the model caches each token’s keys and values so step 500 doesn’t redo steps 1–499.
How it actually works
Autoregressive generation would be wildly wasteful if each new token recomputed attention over the entire history. The KV cache stores the keys and values of all previous tokens so each step only computes the new one and attends against the cache. The trade-off is memory: the cache grows with sequence length and batch size, which is exactly what GQA and MQA aim to tame.
For product teams
The reason long conversations get memory-hungry — and a big driver of serving cost.
For engineers
Cached per-token keys/values reused across decoding steps, trading memory for avoided recomputation.
Related
- Query, Key, Value — Caches attention’s keys and values.
- Grouped-Query Attention — Shrunk by grouped-query attention.
- Inference — A cost during inference.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome