Sparse model
A model that switches on only the parameters it needs per token, so it can be huge but cheap to run.
Think of it like
A big hospital where each patient sees just the right specialists, not every doctor on staff.
Example
A mixture-of-experts model might hold eight times the parameters of a dense one but only light up a fraction per token.
How it actually works
Sparse models — chiefly mixture-of-experts — route each token to a few “expert” subnetworks instead of the whole thing. That decouples total capacity from per-token compute: more knowledge, similar cost. The catch is added complexity in routing, load balancing, and serving.
For product teams
The trick behind models that are “enormous” yet affordable to run — capacity without proportional cost.
For engineers
Conditional computation (e.g. MoE) activating a subset of parameters per token via a router.
Related
- Dense model — The opposite is a dense model.
- Parameters — Grows total parameters.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome