Load balancing
Distributing incoming requests across multiple servers or GPUs so no single one gets overloaded.
Think of it like
A traffic cop directing cars to different lanes so no lane jams up.
Example
10 inference servers sit behind a load balancer; each new request goes to the least-busy server.
How it actually works
Load balancers can use different strategies: round-robin (simple but dumb), least-connections (fewer requests), least-loaded (measured CPU/memory), or health-check-aware (skip dead servers). At scale, you need geographically distributed load balancing too.
For product teams
Load balancing is how you go from one server to many without users noticing.
For engineers
Use a proven balancer (Nginx, HAProxy, cloud load balancers) and monitor health carefully.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome