Throughput
How many requests you can serve per second—measuring capacity and cost efficiency.
Think of it like
Like how many customers a cashier can serve per hour.
Example
Your inference server handles 1000 requests per second. Throughput is 1000 RPS.
How it actually works
Throughput depends on batch size, model size, hardware, and latency. Larger batches increase throughput but increase latency variance (some users wait longer). You optimize for throughput when cost matters (cloud compute is expensive) or when you have bursty traffic. Tradeoff: throughput vs. latency.
For product teams
Throughput directly affects your infrastructure costs. 2x throughput can halve your bill.
For engineers
Measure throughput with realistic request distributions; don't assume uniform arrival rates.
Related
- Latency — Related metric.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome