Needle in a haystack
A test that hides one specific fact in a huge context to see if the model can find it.
Think of it like
Like slipping a secret sentence into a long novel and quizzing the reader on it afterward.
Example
Evaluators bury “the passcode is 4417” deep in 100k tokens of filler, then ask for the passcode to measure recall by position.
How it actually works
The needle-in-a-haystack test probes whether a long context window is truly usable. By varying where the “needle” sits, it maps out weak spots — often the middle — and separates advertised context length from effective context. Passing a simple version doesn’t guarantee handling multiple or reasoning-dependent needles.
For product teams
A quick reality check on long-context claims before you rely on them.
For engineers
Retrieval probe inserting a target fact at varied depths in long context to measure positional recall.
Related
- Lost in the middle — The weakness it exposes.
- Effective context — What it really measures.
- Evals — The broader practice of testing models.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome