Model compression
Reducing model size, memory, and/or compute requirements without losing too much performance.
Think of it like
Packing more into a suitcase without breaking anything.
Example
A 70B model compressed to 13B via distillation, or to 25GB via quantization.
How it actually works
Compression covers multiple techniques: quantization, pruning, distillation, knowledge transfer. Different techniques work for different models and constraints. You might quantize for memory reduction and distill for latency reduction. The key: measure what matters (latency, memory, accuracy) and optimize for all three.
For product teams
Compression unlocks serving at scale. It's not optional for large models.
For engineers
Profile your bottleneck (memory? latency? compute?). Choose compression technique accordingly.
Related
- Umbrella term for efficiency.
- Quantization is one approach.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome