Tensor Parallelism
Splitting a single layer’s math across several GPUs so they compute one big matrix multiply together.
Think of it like
Like four cashiers ringing up different halves of one giant order at the same time, then combining the totals.
Example
Serving a model too big for one GPU often splits each layer’s weight matrices across 4 or 8 GPUs.
How it actually works
Within a layer, the big weight matrices are chopped into pieces spread across GPUs; each computes its slice and the results are combined with fast interconnect communication. It lets a model whose weights don’t fit on one chip run across several as if it were one. The cost is heavy inter-GPU chatter, so it wants very fast links between the chips.
For product teams
A core reason big models need tightly-networked multi-GPU servers to run.
For engineers
Partitioning weight matrices within layers across devices, with all-reduce/all-gather to combine.
Related
- Model Parallelism — The broader family of splitting techniques.
- Pipeline Parallelism — The sibling that splits layers instead.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome