Gating Network
The little module that scores the experts and decides which ones handle each token.
Think of it like
Like a maitre d’ reading the room and deciding which chef should cook each order.
Example
When an MoE layer picks its top two experts for a token, the gating network produced those scores.
How it actually works
It takes the token’s representation and outputs a weight for each expert, and the top few weights decide who runs and how much their output counts. Training has to keep it from favoring the same experts every time, usually with load-balancing penalties. Small as it is, the gating network shapes whether the whole mixture works or collapses onto a handful of overworked experts.
For product teams
A tiny component whose quality quietly determines whether a big sparse model behaves.
For engineers
A learned softmax-scored selector assigning per-token weights to experts, with load-balancing losses.
Related
- Expert Routing — The decision it performs.
- Router Network — Another name for essentially the same part.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome