Model Parallelism
Spreading one model across many GPUs when it’s simply too big to fit on a single chip.
Think of it like
Like a mural too large for one wall, painted across several walls that together make the whole picture.
Example
A frontier model with hundreds of billions of parameters can’t fit on one GPU, so it’s split with model parallelism.
How it actually works
It’s the umbrella term for dividing a model’s parameters and computation across devices, whether by splitting within layers (tensor), across layers (pipeline), or across the sequence (context). It contrasts with data parallelism, where each GPU holds a full copy and just processes different examples. Modern training stacks combine several of these at once.
For product teams
The general reason frontier models demand racks of interconnected GPUs.
For engineers
Distributing model parameters/compute across devices; contrast with data parallelism.
Related
- Tensor Parallelism — A within-layer form.
- Pipeline Parallelism — An across-layer form.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome