Retrieval Latency
Also called Retrieval Latency
The time between asking and getting search results back — before the model even starts answering.
Think of it like
The wait for the waiter to walk to the kitchen and back, separate from how long the meal takes to cook.
Example
A RAG app feels sluggish not because the model is slow but because the vector search over ten million chunks takes 800ms every query.
How it actually works
In a retrieval pipeline, total response time is retrieval plus generation. Retrieval latency depends on index size, the number of results requested, whether you re-rank, and how many retrievers you fan out to. It’s often the hidden tax on fusion and hierarchical setups. You trade it against quality: more candidates and heavier re-ranking find better context but cost milliseconds the user feels.
For product teams
A big driver of "why does this feel slow" that has nothing to do with the model.
For engineers
Retrieval time = f(index size, top-k, rerank, fan-out); budget it separately from generation latency in the end-to-end SLO.
Related
- Semantic Cache — Caching is a main way to cut it.
- Reranking — Re-ranking adds to it for quality.
- Fusion Retrieval — Fusion multiplies the searches involved.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome