Activation quantization
Quantizing the intermediate values (activations) inside a forward pass, not just the weights.
Think of it like
Like cleaning up not just your filing cabinet but also the sticky notes on your desk.
Example
Quantizing both weights and KV cache in an LLM inference, saving memory and compute.
How it actually works
Weights are static and you can calibrate quantization offline. Activations vary per input, so you need dynamic quantization or per-batch calibration. KV cache quantization (quantizing attention keys and values) is especially valuable for long-context inference.
For product teams
Activation quantization can push down memory even further when weight quantization alone isn't enough.
For engineers
Activation quantization is trickier than weight quantization; use mature libraries or expect calibration headaches.
Related
- Model quantization — Converting weights to lower precision.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome