Decoder. plain-English AI glossary

Distributed Inference

▲ Rising

Split model across multiple GPUs or machines so it fits in VRAM and requests parallelize across devices.

Think of it like

A large jigsaw puzzle split across four tables so you can assemble it faster with four people.

Example

70B model will not fit on one A100; shard across two A100s using tensor parallelism; request takes same latency but you serve 2x throughput.

How it actually works

Tensor parallelism splits weight matrices (each GPU holds rows of weight); pipeline parallelism splits layers (GPU0 has layers 1-20, GPU1 has layers 21-40). Communication overhead (all-reduce, all-gather) can negate speedup if link is slow. Best on NVLink or InfiniBand; bad on 1Gbps Ethernet.

For product teams

Enables serving of massive models that do not fit on single GPU.

For engineers

Model partitioned during load; forward pass communicates between GPUs; orchestrator (vLLM, TensorRT-LLM) handles communication patterns.

Model Parallelism — Opposite of.

Read anything AI without the jargon

Look up any term in plain English, or save terms as you read with the free Chrome extension.

Open Decoder Add to Chrome