Latency
How long it takes for the model to produce an output after you send it a request.
Think of it like
Like the time between calling a waiter and getting your food.
Example
You send a query to the model at 12:00:00. You get a response at 12:00:05. Latency is 5 seconds.
How it actually works
Latency is critical for user experience. Anything over a few hundred milliseconds feels slow. It's determined by model size, batch size, hardware, and decoding strategy. You can reduce latency by quantizing the model, using knowledge distillation, or serving smaller models. Tradeoff: latency vs. quality.
For product teams
Every 100ms of additional latency can drop user engagement. Measure latency in production, not just benchmarks.
For engineers
Track p50, p95, and p99 latencies; set SLOs and monitor for regression.
Related
- Throughput — Related metric.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome