Load Balancing Loss
An extra nudge during training that stops a mixture-of-experts model from sending every question to the same few experts.
Think of it like
A manager who notices three staff are swamped while the rest sip coffee, and starts routing tickets more evenly.
Example
A team trains a sparse model and finds two experts handle 80% of tokens; they add a load-balancing term so the idle experts actually learn something.
How it actually works
In a mixture-of-experts layer, a router picks which experts see each token. Left alone, the router collapses onto a favorite few — the rest get no gradient and rot. The load-balancing loss penalizes uneven routing, gently pushing traffic across all experts. Set it too strong and you hurt quality; too weak and experts starve.
For product teams
It is the difference between paying for a big expert model and actually using all of it.
For engineers
An auxiliary loss term proportional to the imbalance between per-expert token counts and their routing probabilities.
Related
- Mixture of Experts — The architecture it exists to keep healthy.
- Expert Parallelism — It regulates the routing that expert parallelism spreads across chips.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome