Request queue
A buffer of pending requests waiting to be processed — long queues mean slow response times.
Think of it like
A line at the grocery store — the longer the line, the longer you wait.
Example
Under load, 500 requests per second arrive but the server can only handle 400/s, so the queue grows and responses get slower.
How it actually works
Queue depth is a key metric. If it's growing, the system is overloaded. If it's consistently long, you need to scale. Queue depth also affects latency perception — users see tail latency spike when the queue builds up.
For product teams
Queue depth is a leading indicator of whether you need to scale or adjust pricing.
For engineers
Monitor queue depth; when it grows, scale immediately (or shed load via rate limiting).
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome