Memory bandwidth
How fast data can be moved from memory to the compute unit — often the bottleneck in LLM inference.
Think of it like
The width of a highway — a wider highway lets more cars through per second.
Example
A GPU with 2 TB/s bandwidth can load more weights per cycle than one with 500 GB/s.
How it actually works
Modern GPUs have massive compute (trillions of FLOPs) but limited bandwidth (hundreds of GB/s). For LLMs, loading weights is often slower than the math itself — you're memory-bound, not compute-bound. Quantization, batching, and optimized kernels all help because they either use less memory or reuse loaded data more.
For product teams
Memory bandwidth is why inference is cheaper than training — it's the limiting resource, not raw compute.
For engineers
Measure compute-bound vs memory-bound; optimize accordingly (fused kernels, better batch sizes).
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome