Mixture of Experts
Also called MoE
A model architecture that routes each input to a few specialist sub-networks instead of using the whole model every time — big capacity, smaller cost.
Think of it like
A hospital with many specialists on staff, but each patient only sees the two or three they need.
Example
Mixtral and GPT-4 use MoE — trillions of total parameters, but only a fraction activate per token.
How it actually works
A router network decides which "expert" sub-networks handle each token. The model can have enormous total parameters (knowledge capacity) while only running a small subset per forward pass (cost). The tradeoff: training is harder, and load-balancing across experts is its own research problem.
For product teams
Architecture that makes giant models affordable to run — big capacity without proportional cost increase.
For engineers
Sparse gating routes tokens to k-of-n expert FFNs; high total params, low per-token compute.
Related
- Parameters — A way to scale models without scaling inference cost linearly.
- Transformer — Still a transformer under the hood.
- Inference — Keeps inference cost manageable.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome