Batching
Waiting for multiple requests to arrive, then processing them together instead of one at a time.
Think of it like
A bus waiting for 30 passengers before leaving, instead of driving immediately with one person.
Example
Your inference server holds requests for 100ms. During that time, 32 requests arrive. You process all 32 together.
How it actually works
Batching amortizes overhead and uses hardware efficiently. But it increases latency for requests that arrive early in the batch window — they wait for later requests. This is the classic throughput-vs-latency tradeoff. You tune the batch window and max batch size for your workload.
For product teams
Batching makes your hardware work for you. Smaller batches = more serving cost.
For engineers
Implement queue-based batching. Tune max batch size and timeout. Monitor for queue buildup under load.
Related
- Efficiency technique.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome