Sparse Attention
Attention that only connects each token to a chosen subset of others instead of all of them.
Think of it like
Instead of everyone at a huge conference talking to everyone, people only chat with their nearest neighbors and a few leaders.
Example
For a very long document, a model attends to nearby tokens plus a few global ones rather than the full quadratic grid.
How it actually works
Full attention costs grow with the square of sequence length, which gets brutal for long inputs. Sparse attention restricts each token to a limited pattern of connections — local windows, strided jumps, a handful of global tokens — cutting the cost while trying to preserve the connections that matter. The art is choosing a sparsity pattern that keeps quality high.
For product teams
A key enabler of long-context models that would otherwise be too expensive to run.
For engineers
Attention restricted to a structured subset of token pairs to reduce quadratic cost.
Related
- Attention — A cheaper form of attention.
- Sliding Window Attention — Relatives include sliding window attention.
- Context length — Helps with long context.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome