Feed-Forward Network
Also called FFN, MLP
The little per-word processing block that sits after attention in every transformer layer.
Think of it like
After a meeting where everyone shares notes, each person goes to their desk to think it over alone.
Example
Following each attention step, the FFN transforms every token’s vector independently before the next layer.
How it actually works
Each transformer layer has two parts: attention mixes information across tokens, then a feed-forward network processes each token on its own through a couple of linear layers with a nonlinearity between them. The FFN holds a large share of the model’s parameters and is where much of its stored knowledge is thought to live. Mixture-of-experts models make this block sparse.
For product teams
Where much of a model’s raw “knowledge capacity” sits — and a big chunk of its parameter count.
For engineers
A position-wise two-layer MLP with a nonlinearity, applied identically to each token after attention.
Related
- Attention — Pairs with attention in each layer.
- Transformer — A building block of the transformer.
- Activation function — Uses an activation function.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome