Expert Parallelism
Putting different experts of a mixture-of-experts model on different chips so they run at the same time.
Think of it like
Splitting a big kitchen into stations — grill, salad, dessert — each on its own cook, plates flying in parallel.
Example
A lab serving a 100-expert model places ten experts per GPU across ten GPUs, so a token only travels to the chip holding its chosen expert.
How it actually works
Unlike splitting one dense layer across chips, expert parallelism assigns whole experts to devices. The catch is the all-to-all shuffle: every token must be shipped to wherever its expert lives and the result shipped back, so network bandwidth, not math, often becomes the bottleneck. Uneven routing makes it worse — one overloaded chip stalls the rest.
For product teams
It is what makes trillion-parameter models affordable to serve without every chip holding every weight.
For engineers
A model-parallel scheme that shards experts across devices, requiring all-to-all communication to route tokens and gather outputs.
Related
- Mixture of Experts — The architecture being spread across hardware.
- Load Balancing Loss — Uneven load turns the all-to-all into a stall.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome