Context extension
Also called Context Window Extension
Stretching a model to handle much longer inputs than it was originally trained for.
Think of it like
Adding leaves to a dining table so it seats far more guests than the original build allowed.
Example
A model trained at 8k tokens is extended to 128k so it can read whole books, using adjusted positional encodings and a bit of long-context fine-tuning.
How it actually works
Models learn positional patterns only up to their training length, so naively feeding more tokens breaks them. Extension techniques modify how positions are encoded — for example interpolating rotary embeddings — often followed by fine-tuning on longer sequences. It works, but longer context is not free: attention cost grows and models often use the far reaches of a long window unevenly.
For product teams
How a model gains the ability to handle long documents without retraining from scratch.
For engineers
Adapt positional encodings (e.g. RoPE interpolation) plus long-sequence fine-tuning to extend usable context.
Related
- Context length — The limit it raises.
- Long-context training — The fine-tuning phase it relies on.
- Positional encoding — The mechanism it modifies.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome