Sparse Mixture
Also called Sparse Mixture of Experts
A mixture-of-experts model where only a few experts fire per token, keeping a giant model cheap to run.
Think of it like
Like a huge consulting firm where any one question only pulls in two specialists, not the entire staff.
Example
A model with 100 billion total parameters but only 12 billion active per token is a sparse mixture.
How it actually works
“Sparse” means each token activates just a small subset of the experts, so total capacity can be enormous while per-token compute stays modest. This decouples a model’s knowledge (total params) from its running cost (active params). The trade-offs are memory to hold all the experts and the routing quality that decides which few actually run.
For product teams
How a model can be “huge” in capability yet still affordable per token.
For engineers
An MoE with top-k expert activation, decoupling total from active parameters.
Related
- Mixture of Experts — The general model class.
- Dense Transformer — The always-on opposite.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome