Pipeline Parallelism
Putting different layers of the model on different GPUs, like stations on an assembly line.
Think of it like
Like a car factory where each station adds one part — the car moves down the line and every station stays busy on a different vehicle.
Example
A 100-layer model might put layers 1–25 on GPU A, 26–50 on GPU B, and so on, passing activations along.
How it actually works
The model is cut into consecutive stages, each on its own device, and data flows through them in sequence. To keep every stage busy rather than idle-waiting, batches are split into micro-batches that flow through in a staggered pipeline. The tricky bit is the “bubble” of idle time at the start and end, which good scheduling minimizes.
For product teams
One of the standard ways to spread a model too large for a single GPU.
For engineers
Partitioning layers into sequential stages across devices, overlapped via micro-batching.
Related
- Model Parallelism — The broader family it belongs to.
- Tensor Parallelism — The sibling that splits within a layer.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome