Sliding Window Attention
Each token only attends to a fixed window of nearby tokens, keeping long sequences affordable.
Think of it like
Reading with a ruler under the line — you focus on the words right around you, not the whole page at once.
Example
A model uses a 4,000-token window so processing a 100,000-token document stays linear instead of exploding.
How it actually works
Sliding window attention limits each token’s view to a local neighborhood of fixed size, making cost grow linearly with length rather than quadratically. Information still travels far because stacking layers lets windows overlap and propagate — like ripples reaching across the sequence over several hops. Many long-context models pair it with a few global tokens for the rest.
For product teams
A practical way to stretch context length without the cost blowing up.
For engineers
Local attention over a fixed-size window per token, giving linear cost with multi-layer receptive-field growth.
Related
- Sparse Attention — A form of sparse attention.
- Attention — Trades global reach for efficiency in attention.
- Context length — Enables longer context.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome