Semantic Cache
Also called Semantic Cache
Reuse a past answer when a new question means the same thing, even if it’s worded differently.
Think of it like
A help desk that recognizes "my card was declined" and "payment won’t go through" as the same ticket and pulls the same answer.
Example
One user asks "how do I reset my password?" and another asks "I forgot my login" — the second gets the cached response instantly, no model call.
How it actually works
A normal cache only hits on exact string matches. A semantic cache embeds the incoming query and checks whether a stored query is close enough in meaning to reuse its answer. It saves latency and API cost on repetitive questions. The danger is the similarity threshold: set it loose and you’ll serve a confidently-wrong cached answer to a subtly different question.
For product teams
Big cost and speed wins on FAQ-style traffic, if you tune the match threshold conservatively.
For engineers
Embed queries; on a near-neighbor hit above a threshold, return the cached response instead of calling the model. Threshold governs the false-hit risk.
Related
- Prompt Caching — Its exact-match cousin for token reuse.
- Embeddings — Matches on meaning via vectors.
- Retrieval Latency — A loose threshold trades correctness for speed.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome