Normalization
Rescaling numbers to a sane, consistent range so training doesn’t wobble or blow up.
Think of it like
Adjusting every photo to the same brightness before comparing them, so lighting doesn’t fool you.
Example
Deep networks add normalization between layers to keep the signal from exploding or vanishing as it goes deep.
How it actually works
Activations can drift to wildly different scales, which makes gradients unstable. Normalization re-centers and re-scales them, keeping the network in a well-behaved zone so it trains faster and deeper. The main variants differ in what they average over.
For product teams
Invisible plumbing, but it’s a big reason very deep models train at all.
For engineers
Standardizing activations (subtract mean, divide by std) to stabilize gradients; see LayerNorm, BatchNorm.
Related
- Layer normalization — The transformer favorite is layer normalization.
- Batch normalization — The older vision trick is batch normalization.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome