Long-Context Architecture
Design choices that let a model read hundreds of thousands of tokens without its cost or memory exploding.
Think of it like
Like redesigning a library so you can pull from a whole wing at once, not just the shelf beside your desk.
Example
Models that accept an entire codebase or a long book in one prompt rely on long-context architecture tricks.
How it actually works
Plain attention cost grows with the square of the sequence length, so naively long contexts become impossibly expensive. Long-context designs use sparse or windowed attention, better positional schemes, cheaper layer types, and specialized caching to keep it tractable. The hard part isn’t just fitting the tokens — it’s making the model actually use information buried in the middle.
For product teams
What lets your product paste in whole documents instead of chunking everything.
For engineers
Architectural techniques (sparse attention, position schemes, caching) mitigating quadratic attention cost.
Related
- Lost in the middle — The failure it fights against.
- Sliding Window Attention — A cheaper attention pattern it may use.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome