Residual Connection
Also called Skip Connection
A shortcut that adds a layer’s input to its output, so information can skip past if needed.
Think of it like
Keeping a photocopy of your draft before edits, so nothing important gets lost in revision.
Example
In each transformer block, the input is added back to what attention produced, keeping gradients healthy in deep stacks.
How it actually works
Very deep networks are hard to train because signals and gradients fade as they pass through many layers. A residual connection adds a layer’s input straight to its output, giving information a clear path forward and gradients a clear path back. This one trick is what made training networks with dozens or hundreds of layers practical.
For product teams
A quiet structural trick that made very deep, capable models trainable in the first place.
For engineers
An additive identity shortcut around a sublayer that preserves signal and eases gradient flow.
Related
- Skip Connection — Also called a skip connection.
- Layer normalization — Paired with layer normalization.
- Deep learning — Essential to deep networks.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome