Decoder. plain-English AI glossary

Paged Attention

▲ Rising

Virtual memory for KV cache: pages of cache are allocated and freed dynamically, avoiding fragmentation and wasted VRAM.

Think of it like

Instead of allocating one huge contiguous block for each request, use small pages that the OS can juggle.

Example

Request needs 256MB KV cache; allocate 256 4MB pages, not a single 256MB block; other requests can use freed pages without fragmentation.

How it actually works

Classic KV cache allocation wastes 90% of memory due to fragmentation (sequences end at different lengths). Paged attention treats cache as virtual memory: pages can be non-contiguous, oversubscribed, evicted. vLLM pioneered this for LLM inference. Reduces effective VRAM requirements by 10x for some workloads.

For product teams

Dramatically improves serving throughput and cost efficiency by reducing memory waste.

For engineers

Kernel allocates cache pages on demand; scheduler pages KV for sequences; can oversubscribe VRAM using host memory.

KV Cache — Optimizes.
VRAM — Reduces.

Read anything AI without the jargon

Look up any term in plain English, or save terms as you read with the free Chrome extension.

Open Decoder Add to Chrome