ALiBi
Also called Attention with Linear Biases
Add a small distance-based penalty to attention scores so far-away tokens matter a little less.
Think of it like
A conversation where voices from across the room come through fainter the farther away they are.
Example
A model trained with ALiBi on short sequences still works on much longer ones at inference without retraining.
How it actually works
Instead of adding position vectors, ALiBi subtracts a penalty from each attention score proportional to how far apart the two tokens are. This bakes in a “nearer matters more” prior and, notably, lets models generalize to sequences longer than those seen in training. It was influential for long-context work, though rotary embeddings have become the more common default.
For product teams
An early trick for length generalization, now largely overtaken by rotary embeddings.
For engineers
Position handling via a linear distance penalty added to attention logits, aiding length extrapolation.
Related
- Rotary Embeddings — An alternative to rotary embeddings.
- Positional Embedding — A kind of positional embedding.
- Attention — Biases attention.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome