Context window management
Strategically managing what information fits in the model's context window — deciding what to include, what to summarize, what to discard.
Think of it like
Packing a suitcase with limited space: you prioritize essentials.
Example
A conversation has 10 prior turns but only room for 5. Summarize the first 5 into "The user asked about X, then Y, we decided on Z."
How it actually works
Context window size is fixed per model. You must allocate space to system message, examples, input, and history. As context fills, you lose memory. Solutions: (1) Summarize old turns, (2) Use external memory, (3) Use models with longer contexts, (4) Slide window (keep only recent turns).
For product teams
Token cost scales with context size. Smart management = lower cost.
For engineers
Track token usage. Implement summarization or sliding window. Choose models with sufficient context for your use case.
Related
- Token limit — Related to token limits and truncation.
- Prompt compression — Addressed by prompt compression.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome