MoE Routing
The overall strategy for deciding which experts each token visits and keeping the load balanced across them.
Think of it like
Like a dispatcher assigning delivery jobs to drivers — route by expertise, but don’t leave some drivers idle and others swamped.
Example
A model card might report “top-2 routing with a load-balancing loss” to describe its MoE routing scheme.
How it actually works
Routing covers both the per-token choice (which top-k experts) and the system-level concern of spreading tokens evenly so no expert is overwhelmed or neglected. Poor routing causes wasted capacity, unstable training, and “dead” experts. Techniques like auxiliary balancing losses and capacity limits exist mostly to keep routing healthy at scale.
For product teams
The design detail that decides whether a sparse model’s promised efficiency actually materializes.
For engineers
The policy and losses governing top-k expert selection and load balancing in MoE layers.
Related
- Router Network — The component that executes it.
- Sparse Mixture — The model class it serves.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome