Throughput Optimization
Maximizing the number of requests a server can handle per second by optimizing batching, memory, and compute.
Think of it like
Fitting more passengers on a bus by optimal seating vs wasted aisle space.
Example
Batch size 128 achieves 1000 req/sec; batch size 64 achieves 600 req/sec; batch size 256 hits OOM—optimal is 128.
How it actually works
Throughput limited by either compute (FLOPS fully utilized, waiting for more data) or memory (cannot fit larger batch). Continuous batching and KV cache optimization critical for LLMs. Hardware choice matters: tensor cores, memory bandwidth. Software profiling identifies bottleneck. Diminishing returns: doubling batch size does not double throughput.
For product teams
Higher throughput = lower cost-per-request = better economics.
For engineers
Profile inference time across batch sizes; optimize memory layout and compute kernels; measure utilization and identify bottleneck (compute vs memory bound).
Related
- Latency — Related to.
- Throughput — Measured by.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome