Effective context
How much of a model’s advertised context window it can actually use well, which is usually less.
Think of it like
Like a phone plan’s “unlimited” data that quietly throttles after a point — the number on the box isn’t the number you get.
Example
A model sold as 128k tokens stays sharp to maybe 32k, then starts missing details buried deeper — its effective context is smaller than its limit.
How it actually works
Maximum context length is a hard ceiling; effective context is the practical range where recall and reasoning hold up. Attention biases like lost-in-the-middle mean quality degrades before the ceiling. Needle-in-a-haystack tests are how you estimate it, and it’s often a fraction of the advertised size.
For product teams
Budget for the usable window, not the marketing number, or long-context features will disappoint.
For engineers
The context range over which recall/reasoning stay reliable, typically below the max token limit.
Related
- Token limit — The hard ceiling above it.
- Lost in the middle — Why it’s smaller than the max.
- Needle in a haystack — How it’s measured.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome