INT4
Storing numbers as 4-bit signed integers, allowing 16 possible values.
Think of it like
A nibble — half a byte.
Example
Instead of a 1-byte int8 weight, use 4 bits, cramming two weights per byte.
How it actually works
INT4 is aggressive — 8x compression compared to float32. But the accuracy loss is significant unless you do quantization-aware training. Decoding int4 requires unpacking, which can be slower than int8. Used in QLoRA and similar parameter-efficient fine-tuning.
For product teams
Extreme compression for resource-constrained settings.
For engineers
Use with QAT or methods like GPTQ. Monitor for severe accuracy loss.
Related
- Aggressive quantization.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome