Prompt Caching
Also called Prompt Caching
Let the model remember the processing of a long, unchanging prefix so it doesn’t redo the work every call.
Think of it like
A chef who preps the same base sauce once in the morning instead of chopping the onions fresh for every single order.
Example
A system prompt with 5,000 tokens of instructions and examples gets cached, so each new user question reuses that prep and returns cheaper and faster.
How it actually works
When many requests share a big identical prefix — a long system prompt, a document, few-shot examples — reprocessing it every time is wasted compute. Prompt caching stores the model’s internal state for that prefix and reuses it, charging less for cached tokens and cutting latency. It only helps when the prefix is byte-for-byte stable and comes first; anything before the cache point invalidates it.
For product teams
Meaningful cost and latency savings for apps with long, repeated system prompts or shared documents.
For engineers
Cache the model’s computed state (KV) for a stable prefix; reuse it across requests that share that exact prefix, billed at a reduced rate.
Related
- KV Cache — The internal state being reused.
- Semantic Cache — Its meaning-based cousin.
- Retrieval Latency — Directly reduces per-call latency.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome