Memory bound
A workload is memory-bound when moving data is slower than processing it — the hardware spends time waiting for data.
Think of it like
A fast reader at a library with only one slow printer — printing takes forever even though the reader is quick.
Example
LLM inference is often memory-bound — loading model weights from memory is slower than the forward pass math.
How it actually works
Memory-bound is common in LLMs because the arithmetic intensity (math per byte loaded) is low — you load huge weight matrices but do relatively little math per weight in inference. Batching helps (you reuse weights across requests), as does quantization (fewer bytes to load) or kernel fusion (keep data in cache between operations).
For product teams
Memory-bound inference means cost-per-token is limited by bandwidth, not chip power.
For engineers
To optimize memory-bound workloads: reduce data size (quantization), fuse kernels, or increase batch size.
Related
- Compute bound — When compute is the bottleneck instead.
- Memory bandwidth — The limiting data throughput.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome