What Is Quantization?
Quantization is the practice of running a model at lower numerical precision to increase throughput and cut memory and cost, accepting a controlled tradeoff in accuracy. Weights, activations, and the KV cache can each be quantized, and they are often quantized to different formats within the same deployment.
Common formats run from FP16 and BF16 down to FP8, FP4, and integer types like INT8. Lower precision means fewer bytes moved per token, which is what produces the speedup: LLM decode is memory-bandwidth bound far more often than it is compute bound.
Precision Is Hardware-Gated
Which formats are available is a property of the silicon, not the model:
| Format | Requires |
|---|---|
| FP16 / BF16 | Ampere-class and newer |
| FP8 | Hopper-class or newer |
| FP4 | Blackwell |
| INT8 | Broadly supported |
A precision path that is available on a B200 may not exist on an A100. This is one reason the same model has different economics on different generations of hardware, and why a quantization plan has to be written against the fleet an operator actually runs.
The Tradeoff
Done well, quantization is one of the largest levers on cost per token. Done carelessly, it is a common cause of quality shortfalls. Aggressive quantization combined with poor sampling defaults degrades endpoint accuracy even when the speed numbers look good, and the degradation often does not show up until it hits a specific class of prompt in production.
The practical guard is to hold quantized endpoints to an accuracy benchmark, not just a throughput benchmark, and to treat the quantization format as part of what an endpoint publishes rather than an internal detail.
