Utilization
The percentage of the GPU/CPU's total capacity that's actually being used — idle hardware is wasted money.
Think of it like
How full a delivery truck is when it leaves — 100% utilization means no empty space.
Example
A GPU running inference at 60% utilization means 40% of its compute power is sitting idle.
How it actually works
High utilization is the goal — it means you're squeezing value from your hardware. For inference, this usually means batching (more requests) or larger models. For training, it often means larger batch sizes. Low utilization usually signals memory pressure (can't fit bigger batches) or architectural bottlenecks.
For product teams
High utilization = cheaper cost-per-inference; low utilization = overpaying for idle hardware.
For engineers
Monitor utilization and profile bottlenecks; if it's memory, try quantization; if it's waiting, try batching.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome