GPU
A specialized processor optimized for parallel computation, essential for training and serving neural networks.
Think of it like
A specialized worker who's really fast at doing lots of simple calculations in parallel.
Example
An Nvidia A100 GPU can do trillions of floating-point operations per second, making it 100x faster than a CPU for matrix multiplication.
How it actually works
GPUs excel at matrix operations because they have thousands of cores running in parallel. This is perfect for neural networks. The tradeoff: GPUs are expensive ($5k–$20k), use lots of power, and have limited memory (40–80GB for consumer GPUs). Model parallelism, data parallelism, and pipelining are techniques to work around these limits.
For product teams
GPU cost dominates the serving budget for large models.
For engineers
Profile your model. Estimate memory (parameters × bytes per param). Choose GPU based on memory and latency needs.
Related
- The hardware.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome