Positional encoding
The trick that tells a model the order of words, since the core machinery would otherwise see your sentence as an unordered pile.
Think of it like
Numbering the pages so the story still makes sense after you drop the binder.
Example
It’s why “dog bites man” and “man bites dog” land differently instead of looking identical.
How it actually works
Attention treats input as a set, so order has to be injected separately — as a positional signal added to each token. The specific method keeps evolving (sinusoidal, learned, rotary), and it’s a quiet lever behind how far a model’s context window can stretch. Increasingly an under-the-hood detail as newer schemes just work.
For product teams
Deep-internals; matters only when reasoning about very long-context behavior.
For engineers
Injects sequence order into permutation-invariant attention; RoPE and variants affect context extrapolation.
Related
- LLM — A detail of how the transformer behind an LLM works.
- Context window — Bounds how the context window scales.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome