Continuous batching
Processing multiple requests simultaneously at the token level, starting new requests as old ones finish, without waiting for a batch to fill.
Think of it like
A restaurant serving plates as they're done, instead of waiting for all plates to be ready before serving anyone.
Example
Request A is generating token 50 of 100. Request B just arrived. You start processing Request B token 1 while finishing Request A. Once Request A finishes, you start Request C. No idle GPU.
How it actually works
Continuous batching avoids the latency penalty of traditional batching — you don't have to wait for a full batch. It also keeps hardware utilized longer. The downside: implementation is complex (need to manage KV caches for varying lengths) and bugs can cause crashes. Systems like vLLM made it practical.
For product teams
Modern standard for serving. Better latency without sacrificing throughput.
For engineers
Use production systems like vLLM. Track KV cache usage. Test overload handling.
Related
- Advanced serving pattern.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome