What is Quantization?
Quantization is the process of mapping model weights, activations, or KV-cache values from a higher-precision numeric representation to a lower-precision representation with explicit scales, zero points, and granularity. It can reduce storage, memory traffic, or compute cost, but it does not guarantee faster inference or acceptable model behavior.
Quick Facts
| Full Name | Model Quantization |
|---|---|
| Created | Technique from 1990s, popularized for LLMs in 2023 |
| Specification | Official Specification |
How It Works
Model quantization is a deployment technique rather than a universal compression ratio. A complete recipe identifies the tensor target, numeric representation, calibration data, quantization algorithm, granularity, excluded modules, packing layout, runtime, and hardware. Post-training quantization applies a recipe to an existing checkpoint; quantization-aware training simulates low-precision effects during training. GPTQ, AWQ, and SmoothQuant are algorithms with different objectives, while GGUF is a model container and runtimes such as vLLM or llama.cpp execute supported tensor encodings. The theoretical weight payload is only a lower bound because scales, metadata, unquantized layers, activations, KV cache, workspace, and concurrency also consume memory. Production acceptance therefore requires task, safety, latency, memory, and goodput measurements against the exact baseline.
Key Characteristics
- Can target weights, activations, or KV cache independently
- Uses explicit bit width, scale, zero point, and tensor granularity
- Includes post-training quantization and quantization-aware training workflows
- Depends on matching artifact layouts to runtime kernels and hardware
- Requires representative calibration and workload-specific quality evaluation
- Reduces nominal precision without guaranteeing latency or cost improvement
Common Use Cases
- Reducing weight storage and memory traffic for local or server inference
- Increasing useful concurrency when model weights constrain accelerator memory
- Using low-precision matrix kernels on supported inference hardware
- Reducing KV-cache capacity pressure for long-context serving after validation
- Loading a frozen low-precision base model for adapter training workflows
Example
Loading code...Frequently Asked Questions
What is the difference between weight, activation, and KV-cache quantization?
Weight quantization primarily reduces artifact size and weight memory traffic. Activation quantization enables lower-precision computation but must handle runtime outliers and accumulation error. KV-cache quantization targets memory growth from context length and concurrency. A deployment may use one, two, or all three, and each target needs separate quality and performance validation.
Does a four-bit model use exactly one quarter of FP16 memory?
No. Four-bit weights have a theoretical payload equal to one quarter of FP16 weights, but real artifacts also store scales, zero points, group metadata, alignment, and unquantized modules. Resident service memory additionally includes activations, KV cache, runtime workspace, allocator fragmentation, and concurrent requests.
What is the difference between PTQ and QAT?
Post-training quantization converts an existing checkpoint using weights and, for many methods, representative calibration data. Quantization-aware training simulates low-precision behavior during training so parameters can adapt. QAT is not automatically superior; both workflows must be compared as exact deployable artifacts on the target tasks.
Is GGUF a quantization method?
No. GGUF is a binary model container that stores metadata and tensors, including tensors encoded with supported quantized types. The quantization recipe determines how tensor values are approximated, while the container packages them and a compatible runtime executes them.
How should a quantized model be approved for production?
Pin the baseline revision, tokenizer, chat template, quantizer, calibration data, artifact checksum, runtime, kernel, and hardware. Then compare business acceptance, tool calls, structured output, safety slices, long context, TTFT, inter-token latency, peak memory, stable concurrency, and goodput. Promote through shadow and canary stages with a ready baseline rollback.