Context Parallelism
Splitting one very long input across multiple GPUs so no single chip has to hold the whole sequence.
Think of it like
Like several people each reading a different third of a huge document, then comparing notes to understand the whole.
Example
Training a model on a 1-million-token context often splits that sequence across many GPUs via context parallelism.
How it actually works
Attention over very long sequences needs enormous memory, more than one GPU has. Context parallelism shards the sequence dimension across devices, so each holds part of the tokens, and they exchange the pieces of attention that cross boundaries. It’s one of several parallelism strategies that make long-context and large-model training possible at all.
For product teams
Part of the plumbing that makes million-token context windows feasible to train.
For engineers
Sharding the sequence dimension across devices, with communication for cross-shard attention.
Related
- Tensor Parallelism — The sibling that splits the math within a layer.
- Pipeline Parallelism — The sibling that splits layers across devices.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome