Positional Embedding
Extra information that tells the model where each token sits, since attention alone ignores order.
Think of it like
Numbering the pages of a manuscript before shuffling them, so the order can always be recovered.
Example
Without positional info, “dog bites man” and “man bites dog” look identical to attention — embeddings fix that.
How it actually works
Attention is order-blind: it treats its inputs as a set, so without positional signals a transformer can’t tell sequence order. Positional embeddings supply that missing structure, either learned per position or computed by a fixed function, and are added to or folded into token representations. Which scheme you pick strongly affects how well the model handles long or unusual-length inputs.
For product teams
The mechanism that lets an order-blind architecture understand word order at all.
For engineers
Signals encoding token position added to or combined with embeddings to restore sequence order.
Related
- Rotary Embeddings — Modern versions include rotary embeddings.
- ALiBi — And ALiBi.
- Positional encoding — Compare positional encoding.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome