Rotary Embeddings
Also called RoPE
Encode a token’s position by rotating its vector, so relative distances fall out naturally in attention.
Think of it like
Turning each clock hand by an amount set by the hour — the angle between hands tells you the gap.
Example
A model uses RoPE so that “word 10 relative to word 7” is captured cleanly, and it extends to longer inputs gracefully.
How it actually works
RoPE injects position by rotating each token’s query and key vectors by an angle proportional to their position. Because attention compares queries and keys via dot products, these rotations make the score depend on the relative distance between tokens — a clean, position-aware signal. It also extends to longer sequences than seen in training better than older schemes, which is why it’s now common.
For product teams
Part of why modern models handle long inputs and relative word order so well.
For engineers
Position encoding via rotation of query/key vectors, yielding relative-position-aware dot products.
Related
- Positional Embedding — A modern positional embedding.
- ALiBi — An alternative is ALiBi.
- Attention — Shapes attention.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome