Concurrency
How many requests are being processed at the same time — high concurrency means more throughput on the same hardware.
Think of it like
A cashier with 5 checkout lanes running simultaneously is more concurrent than 1 lane.
Example
A server that handles 100 concurrent requests will serve more total requests than one that only handles 10 at a time.
How it actually works
Concurrency is bounded by memory and compute. Each concurrent request needs some memory (KV cache, input/output buffers). Continuous batching lets you mix requests at different completion stages (one nearly done, one just started) for better GPU utilization.
For product teams
High concurrency on the same hardware means more users served per dollar.
For engineers
Use continuous batching and careful memory management to support high concurrency.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome