Model quantization
Converting model weights from high-precision floats (FP32) to lower precision (FP16, INT8) to save memory and speed up inference.
Think of it like
Compressing a FLAC audio file to MP3 — loses some precision but much smaller.
Example
An 8GB model in FP32 becomes 2GB in INT8, runs 2-4x faster, and stays accurate enough for most uses.
How it actually works
Lower precision means fewer bits per weight. FP16 uses 16 bits per float; INT8 uses 8 bits and requires careful scaling. Post-training quantization (apply it after training) is easy but loses more accuracy; quantization-aware training (teach the model during training that it'll be quantized) is better but costs training time.
For product teams
Quantization can cut costs 3-4x by reducing memory and compute without huge accuracy loss.
For engineers
Use libraries like GPTQ or bitsandbytes; measure accuracy on your actual use cases.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome