Mixture of Experts
Also called MoE
Split a model into many specialist sub-networks and route each token to just a few of them.
Think of it like
A hospital where a receptionist sends each patient to the two right specialists instead of every doctor at once.
Example
A model has 64 expert blocks but activates only two per token, so it’s huge in capacity yet cheap to run.
How it actually works
In a mixture-of-experts layer, a lightweight router picks a small subset of expert sub-networks to process each token, so total parameters can be enormous while compute per token stays modest. The wins are capacity and efficiency; the headaches are load-balancing the experts, routing instability, and the memory to hold all those parameters. It powers several frontier models.
For product teams
How models get much bigger in capability without a proportional jump in inference cost.
For engineers
A layer that routes each token to a few of many expert subnetworks, decoupling parameter count from per-token compute.
Related
- Feed-Forward Network — Makes the feed-forward network sparse.
- Sparse model — A sparse model design.
- Dense model — Contrast with a dense model.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome