prompt caching API
An API feature where you cache the prefilled tokens (prompt, system instructions, uploaded documents) and pay less to reuse them in follow-up queries.
Think of it like
Prepping ingredients the night before—do the expensive work once, reuse for multiple meals.
Example
Upload a 100k-token document. Tell the API to cache it. Make 10 queries against the doc. You pay full price for the first query, discount (e.g., 10%) for the rest.
How it actually works
Prompt caching reuses KV cache entries from initial processing. It requires exact match of prefix tokens (so changing system prompt breaks cache). Saves both cost and latency. Relatively new feature; adoption growing.
For product teams
Reduces cost for document-heavy workflows; enables conversation-over-documents at scale.
For engineers
KV cache management; prefix matching; usually 90% discount on cached tokens.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome