GPTQ
A quantization method that uses the Hessian to intelligently decide which weights to quantize less, so you get good accuracy at low bit-widths.
Think of it like
A smart editor who cuts the least important words to make a story fit in fewer pages.
Example
Using GPTQ you can quantize GPT models to 3 bits with minimal accuracy loss, or 4 bits with nearly no loss.
How it actually works
GPTQ computes the Hessian (second-order information) about which weights matter most, then uses that to set per-weight quantization scales. It's slower to apply than simple post-training quantization, but the accuracy payoff is huge. Ollama and text-generation-webui popularized pre-quantized GPTQ models.
For product teams
GPTQ is the standard for distributing small, fast, accurate open-source models.
For engineers
GPTQ models are fast and efficient; libraries like GPTQ-for-LLaMA or Marlin make them easy to use.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome