Long-context training
Also called Long-Context Training
Fine-tuning a model on very long sequences so it can actually use a big context window, not just accept one.
Think of it like
Practicing with a full-length novel, not just short passages, so you can genuinely follow a long story.
Example
After extending a model to 128k tokens, the team trains it on long documents so it learns to track information across the whole span.
How it actually works
Accepting long inputs and using them well are different things. Long-context training exposes the model to genuinely long sequences during fine-tuning so it learns to attend across distances it never saw in pretraining. It is expensive — attention and memory scale with sequence length — and it directly fights the "lost in the middle" tendency where models under-use the center of a long input.
For product teams
The work that turns a big advertised context window into one that actually performs.
For engineers
Fine-tune on long sequences post-extension so attention learns to use the full window; costly in memory/compute.
Related
- Context extension — The step that raises the ceiling first.
- Lost in the middle — The failure it combats.
- Context length — The window it makes usable.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome