Layer normalization
Normalization done across each token’s own features — the version transformers actually use.
Think of it like
Grading each student against their own average across subjects, not against the whole class.
Example
Every transformer block sandwiches its attention and feed-forward steps with layer norm to stay stable.
How it actually works
LayerNorm normalizes across the feature dimension for each token independently, so it doesn’t depend on other examples in the batch. That independence is why it plays nicely with variable-length sequences and small batches, unlike batch norm. It’s a quiet workhorse of the transformer.
For product teams
One of those default design choices you inherit for free in any modern model.
For engineers
Normalizes over the feature axis per position; batch-size-independent, standard in transformers.
Related
- Normalization — A flavor of normalization.
- Layer — Wraps each layer.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome