Batch inference
Processing multiple requests together in one forward pass instead of one at a time, which is much faster per request.
Think of it like
Sending 10 letters in one mail batch instead of 10 separate trips to the post office.
Example
Instead of running the model 100 times on 100 individual prompts, you group them into batches of 32 and run 4 batches — same total compute but much faster wall-clock time.
How it actually works
Batching amortizes the overhead of a forward pass (loading model to GPU, cache management) across multiple requests. Batch size is limited by memory. Small batches waste compute potential; large batches need more memory. Good inference systems balance request throughput with latency via continuous batching (requests arrive and leave asynchronously).
For product teams
Batching is the #1 lever for throughput — more requests per second with the same hardware.
For engineers
Implement continuous batching so requests don't block; use dynamic batch sizes.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome