Caching
Storing computed results so you don't have to recompute them when the same request comes in again.
Think of it like
Writing down yesterday's math homework so you don't have to solve it again today.
Example
If 10 users all ask the same question, the first call generates the answer; the next 9 see the cached result instantly.
How it actually works
Caching happens at multiple levels: KV caching during a single forward pass (storing attention keys and values), prompt caching (reusing embeddings of long context), and application-level caching (storing full responses). Cache misses waste compute; cache hits save both latency and money.
For product teams
Caching is the fastest, cheapest way to speed up responses — if you can predict repeated patterns.
For engineers
Implement cache-key semantics carefully — wrong keys cause stale responses; too many cache layers and debugging becomes hell.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome