Dense model
A model where every parameter fires on every token — the standard, straightforward design.
Think of it like
A restaurant where the whole kitchen cooks every order, versus one that pages only the relevant chef.
Example
A classic dense 7B model uses all 7 billion parameters for each token; a sparse model of the same total wouldn’t.
How it actually works
In a dense model the full weight set participates in every forward pass. It’s simple and predictable but means compute scales directly with size. The alternative, sparse mixture-of-experts, activates only a slice per token to get more capacity for less compute.
For product teams
Predictable cost and behavior; you pay for the whole model on every token.
For engineers
All parameters active per token; compute ∝ parameter count, contrasted with sparse MoE routing.
Related
- Sparse model — The opposite is a sparse model.
- Model size — Cost scales with model size.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome