Attention Sink
The odd habit models have of dumping spare attention onto the very first token, using it as a parking spot.
Think of it like
Like a conversation where everyone reflexively glances at the person at the head of the table, whether or not they’re relevant.
Example
Researchers found that keeping the first few tokens in the cache preserves quality in long streaming generation — because they’re acting as sinks.
How it actually works
Attention weights must sum to one, so when a token has nothing important to focus on, the model learns to offload that leftover attention somewhere harmless — usually the first token. Understanding this fixed a real bug: naively dropping early tokens in long conversations wrecked models until people realized those tokens were load-bearing sinks. It’s a small quirk with outsized practical impact.
For product teams
An obscure detail that quietly governs whether long, streaming chats stay coherent.
For engineers
The tendency of softmax attention to allocate residual mass to initial tokens, aiding long-context stability.
Related
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome