Capacity Factor
A dial that sets how many tokens each expert is allowed to handle before it starts dropping the overflow.
Think of it like
A restaurant setting covers per table — go over and the extra guests wait outside or get turned away.
Example
An engineer raises the capacity factor from 1.0 to 1.25 after noticing tokens were being dropped when too many routed to one expert at once.
How it actually works
Experts run in fixed-size buffers for hardware efficiency, so each has a hard cap: capacity = factor × (tokens ÷ experts). A factor above 1 leaves slack for uneven routing; tokens beyond the cap get dropped or passed through untouched. Higher factor means fewer drops but more wasted compute on half-empty buffers.
For product teams
A quiet quality-versus-cost knob most people never see but that shapes both.
For engineers
The multiplier on the per-expert token buffer size in a MoE layer; overflow tokens are dropped past the cap.
Related
- Load Balancing Loss — The dropping only matters because of how experts are balanced.
- Expert Parallelism — It bounds the buffers that expert parallelism physically allocates.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome