LLM Quantization: Methods, Runtime Fit, and Evaluation

Direct Answer

LLM quantization is not a single compression switch. It is a deployment contract that specifies what is quantized (weights, activations, or KV cache), how values are represented (for example W4A16 or W8A8), which algorithm and calibration data produced the artifact, and which runtime kernel executes it. A lower nominal bit width can reduce storage or memory traffic, but it does not guarantee lower latency, lower total memory, or acceptable model behavior. The only defensible choice is one that passes workload-specific quality and serving gates on the target hardware.

Key Takeaways

• Separate algorithms such as GPTQ, AWQ, and SmoothQuant from numeric formats, model containers such as GGUF, and inference runtimes.
• Estimate weight bytes only as a lower bound. KV cache, metadata, scales, workspace, allocator behavior, and concurrency determine service memory.
• Treat calibration data, tokenizer, chat template, quantizer version, packing layout, and artifact checksum as part of the model contract.
• Compare candidates in a controlled lane and an ecosystem-optimized lane; do not mix unrelated engines and call the result an algorithm ranking.
• Promote a quantized artifact only when business quality, safety, latency, memory, and goodput all satisfy explicit gates.

What LLM Quantization Changes

Quantization approximates real-valued tensors with a finite set of representable values. A common affine mapping is:

q = clip(round(x / scale) + zero_point, q_min, q_max)
x_hat = scale * (q - zero_point)

The scale and optional zero point can be shared per tensor or calculated per channel, group, or block. Finer granularity usually tracks local distributions more closely, but stores more metadata and may require different kernels.

The notation W4A16 means four-bit weights with 16-bit activations. W8A8 means eight-bit weights and eight-bit activations. These labels still omit important details: group size, symmetric versus asymmetric mapping, accumulator precision, excluded modules, packing layout, and whether the KV cache remains in FP16 or BF16.

Quantization Targets Are Independent

Weights, activations, and KV cache solve different memory and compute problems.

A weight-only artifact does not automatically reduce KV-cache memory. This matters for long prompts and high concurrency, where cache allocation can dominate available memory. See the separate KV cache engineering guide for paging, isolation, and capacity boundaries.

Do Not Compare Algorithms, Formats, and Runtimes as Peers

Quantization discussions often place GPTQ, AWQ, GGUF, bitsandbytes, and llama.cpp in one table. They belong to different layers.

The GGUF specification defines a binary file format with extensible metadata, tensor information, alignment, and memory-mapping support. GGUF can carry tensors encoded with different types, but it is not itself a peer algorithm to GPTQ or AWQ.

Runtime support is also a moving target. The current vLLM quantization documentation maintains method and hardware compatibility rather than promising that every quantized artifact runs efficiently everywhere. Pin the runtime version and verify its current matrix before producing an artifact.

How GPTQ, AWQ, and SmoothQuant Differ

Each method optimizes a different approximation problem. Paper results establish mechanisms under reported experiments; they do not establish a permanent winner for every model and runtime.

GPTQ Uses Approximate Second-Order Information

GPTQ is a post-training, weight-only method derived from one-shot weight quantization. It uses approximate second-order information to choose rounding updates while quantizing a layer. The relevant engineering questions are whether the target model is supported, how calibration samples represent production inputs, and whether the deployed runtime has a kernel for the produced packing layout.

GPTQ paper speed and quality results belong to the models, hardware, software, and settings in that paper. A new deployment must reproduce its own baseline.

AWQ Protects Activation-Salient Channels

AWQ is an activation-aware, weight-only method. It observes activation distributions, identifies salient weight channels, and searches equivalent per-channel scaling that reduces their quantization error without retaining a mixed-precision subset. The method does not require backpropagation or weight reconstruction.

AWQ does not guarantee higher quality or speed than GPTQ. Its paper reports results for a specific implementation and workload. Runtime kernels, model architecture, group size, and calibration representativeness decide the result in production. The dedicated AWQ glossary entry explains this mechanism without turning paper benchmarks into universal claims.

SmoothQuant Moves Activation Difficulty into Weights

SmoothQuant targets W8A8 inference. It applies an equivalent transformation that smooths activation outliers by migrating quantization difficulty from activations into weights. This makes both operands friendlier to integer matrix multiplication, but the migration strength and runtime implementation remain deployment choices.

Why Bit Width Does Not Equal Service Memory

The theoretical weight payload is a useful lower bound, not a capacity plan:

ideal_weight_bytes = parameter_count * bits_per_weight / 8

Actual artifact and resident memory add scales, zero points, group metadata, unquantized layers, tokenizer files, alignment, and container overhead. Runtime memory adds KV cache, temporary activations, CUDA graphs or compiled buffers, communication buffers, allocator fragmentation, and request queues.

Therefore, statements such as “a 7B INT4 model needs under 4 GB” are incomplete. They do not specify whether “needs” means file size, loaded weight memory, idle process memory, single-request peak, or safe serving capacity. The LLM inference guide shows why TTFT, inter-token latency, batching, and scheduling must be evaluated together.

Lower Precision Does Not Guarantee Speed

A quantized path becomes faster only when reduced memory traffic or low-precision compute exceeds its overhead. Common failure modes include:

• the runtime dequantizes weights into a wider type before each operation;
• the artifact packing does not match the available kernel;
• unsupported modules fall back to slower kernels;
• short prompts and small batches cannot amortize launch and conversion overhead;
• decode is constrained by KV-cache traffic or scheduling rather than weight reads;
• repacking during startup increases load time and operational variance.

Measure prefill and decode separately. A candidate can improve throughput while worsening time to first token, or reduce weight memory without increasing useful concurrency.

Build a Reproducible Quantization Contract

A quantized artifact should be traceable to an immutable baseline and reproducible recipe. Store the contract beside the artifact, not in a deployment ticket that will disappear.

{
  "artifact_id": "support-model-w4a16-awq-r3",
  "base_checkpoint": "registry.example/model",
  "base_revision": "immutable-revision",
  "tokenizer_revision": "immutable-revision",
  "chat_template_sha256": "sha256:...",
  "quantizer": {
    "name": "awq",
    "version": "pinned-version",
    "weight_bits": 4,
    "activation_bits": 16,
    "group_size": 128,
    "symmetric": false,
    "excluded_modules": ["output_head"]
  },
  "calibration": {
    "dataset_id": "approved-calibration-slice",
    "dataset_sha256": "sha256:...",
    "sampling_policy": "language-and-task-stratified"
  },
  "packaging": {
    "format": "runtime-specific",
    "runtime": "pinned-runtime",
    "kernel": "verified-kernel-family"
  },
  "artifact_sha256": "sha256:...",
  "license": "verified"
}

The calibration slice should represent production languages, prompt lengths, domains, tools, and safety-sensitive inputs. “A few hundred generic samples” is not a universal sufficiency rule. Sample count is meaningful only alongside coverage and stability tests.

Keep the tokenizer and chat template fixed. A template change can alter output quality more than quantization and invalidate attribution.

Benchmark in Two Lanes

A fair evaluation separates method quality from ecosystem maturity.

Controlled Lane

Use the same baseline checkpoint, revision, tokenizer, chat template, calibration slice, evaluation set, server version, hardware, request schedule, and measurement code. Change only the quantization recipe when possible. This lane answers: what did the quantization choice change under controlled conditions?

Ecosystem-Optimized Lane

Allow each candidate to use its best supported runtime and kernel while keeping model identity, workload, quality gates, and hardware class comparable. This lane answers: what can each deployable stack deliver in practice?

Do not combine tokens-per-second numbers from different engines, batch sizes, prompt lengths, or output lengths into an algorithm leaderboard.

Goodput counts only requests that meet both quality and latency objectives. It prevents a high-throughput but unusable candidate from winning.

A Runnable Candidate Gate

The following dependency-free Python program validates measured candidates. It does not quantize a model; it prevents promotion based on bit width or throughput alone.

from __future__ import annotations

import argparse
import json
from pathlib import Path
from typing import Any


REQUIRED_METRICS = {
    "business_acceptance": (0.0, 1.0),
    "structured_output_validity": (0.0, 1.0),
    "safety_pass_rate": (0.0, 1.0),
    "ttft_p95_ms": (0.0, None),
    "itl_p95_ms": (0.0, None),
    "peak_memory_gib": (0.0, None),
    "goodput_rps": (0.0, None),
}


def number(value: Any) -> bool:
    return isinstance(value, (int, float)) and not isinstance(value, bool)


def validate(candidate: dict[str, Any]) -> list[str]:
    errors: list[str] = []
    for field in ("artifact_id", "base_revision", "artifact_sha256"):
        if not isinstance(candidate.get(field), str) or not candidate[field]:
            errors.append(f"{field} must be a non-empty string")

    metrics = candidate.get("metrics")
    gates = candidate.get("gates")
    if not isinstance(metrics, dict) or not isinstance(gates, dict):
        return errors + ["metrics and gates must be objects"]

    for name, (lower, upper) in REQUIRED_METRICS.items():
        value = metrics.get(name)
        if not number(value) or value < lower or (
            upper is not None and value > upper
        ):
            errors.append(f"invalid metric: {name}")

    for name, rule in gates.items():
        if name not in REQUIRED_METRICS:
            errors.append(f"unknown gate: {name}")
            continue
        if not isinstance(rule, dict) or set(rule) != {"op", "value"}:
            errors.append(f"gate {name} needs op and value")
            continue
        op, limit = rule["op"], rule["value"]
        if op not in {"min", "max"} or not number(limit):
            errors.append(f"invalid gate: {name}")
            continue
        measured = metrics.get(name)
        if number(measured):
            passed = measured >= limit if op == "min" else measured <= limit
            if not passed:
                errors.append(
                    f"gate failed: {name} measured={measured} {op}={limit}"
                )
    return errors


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("candidate", type=Path)
    args = parser.parse_args()
    try:
        value = json.loads(args.candidate.read_text(encoding="utf-8"))
    except (OSError, json.JSONDecodeError) as error:
        parser.error(str(error))
    if not isinstance(value, dict):
        parser.error("candidate root must be an object")
    errors = validate(value)
    if errors:
        for error in errors:
            print(f"ERROR: {error}")
        raise SystemExit(1)
    print(f"candidate passed: {value['artifact_id']}")


if __name__ == "__main__":
    main()

Save it as quantization_gate.py, then evaluate a measured candidate:

python quantization_gate.py candidate.json

The input must contain real measurements from a pinned test. A valid shape looks like:

{
  "artifact_id": "support-model-w4a16-awq-r3",
  "base_revision": "immutable-revision",
  "artifact_sha256": "sha256:...",
  "metrics": {
    "business_acceptance": 0.94,
    "structured_output_validity": 0.99,
    "safety_pass_rate": 0.98,
    "ttft_p95_ms": 420,
    "itl_p95_ms": 38,
    "peak_memory_gib": 18.6,
    "goodput_rps": 7.4
  },
  "gates": {
    "business_acceptance": {"op": "min", "value": 0.92},
    "structured_output_validity": {"op": "min", "value": 0.98},
    "safety_pass_rate": {"op": "min", "value": 0.97},
    "ttft_p95_ms": {"op": "max", "value": 500},
    "itl_p95_ms": {"op": "max", "value": 45},
    "peak_memory_gib": {"op": "max", "value": 20},
    "goodput_rps": {"op": "min", "value": 6.5}
  }
}

These values illustrate the schema; they are not claims about a model or method.

Promotion, Canary, and Rollback

Quantized model promotion should follow the same discipline as a model or serving change. Do not replace the baseline fleet immediately after an offline benchmark.

1. Verify manifest completeness and artifact checksum.
2. Run offline quality and performance gates against the exact baseline.
3. Shadow representative traffic without user-visible output.
4. Canary by tenant or workload, not by an untraceable random artifact swap.
5. Monitor quality proxies, tool errors, schema failures, latency, memory, and goodput.
6. Roll back to the pinned baseline when any hard gate fails.

Keep baseline and candidate capacity compatible during the canary. A rollback procedure that requires rebuilding or downloading the old artifact is not an operational rollback.

Common Failure Modes

Selecting by a Universal Ranking

“AWQ for speed, GPTQ for accuracy, GGUF for CPU” is not a durable decision rule. It hides model support, kernel maturity, artifact provenance, and workload shape.

Measuring Only Perplexity

Perplexity can detect some distribution shifts, but it does not prove tool-call correctness, structured output validity, multilingual behavior, refusal policy, or business task acceptance.

Ignoring Calibration Governance

Unversioned calibration data makes the artifact irreproducible. Sensitive production prompts can also create privacy and licensing problems. Record data lineage and use approved, representative slices.

Comparing Different Model Revisions

A quantized candidate built from a different checkpoint, tokenizer, or chat template cannot isolate quantization effects. Pin every component before attributing a quality change.

Treating Successful Loading as Compatibility

A runtime that loads an artifact may still repack it, fall back for some layers, or use a slow kernel. Confirm logs and profiles, then measure the complete service path.

Frequently Asked Questions

Is INT8 always safer than INT4 for production?

No. A wider representation often offers more headroom, but production safety is a measured property, not a bit-width label. An INT4 artifact with a mature kernel and strong task results may be preferable to an INT8 path with unsupported operations or poor capacity. Apply the same quality and service gates to both.

Does quantization reduce cloud cost?

It can reduce cost if memory savings increase useful concurrency or low-precision kernels improve goodput. It can also leave cost unchanged when the workload is queue-bound, KV-cache-bound, or forced onto inefficient kernels. Calculate cost per accepted request at the required SLO rather than cost per raw token.

Is QAT always more accurate than PTQ?

No. Quantization-aware training gives the optimizer an opportunity to adapt, but its result depends on training data, objective, stability, and deployment format. Strong post-training methods can be sufficient, while a poorly controlled QAT run can regress behavior. Compare exact artifacts.

Can the same quantized artifact run in every runtime?

No. Containers, tensor encodings, packing layouts, operators, and kernels are runtime-specific. Use the runtime's current compatibility documentation, pin versions, and verify the loaded kernel path.

Should I quantize before fine-tuning?

It depends on the workflow. QLoRA loads a quantized frozen base while training adapters, which is different from producing a final serving artifact. After adapter merge or composition, evaluate and package the exact artifact that will be served. The LoRA glossary and local LLM deployment guide cover the surrounding workflow.

Summary

LLM quantization is an end-to-end serving decision, not a universal compression ratio. Separate the tensor target, numeric representation, quantization algorithm, container, and runtime. Preserve the complete artifact contract, compare candidates under controlled and ecosystem-optimized lanes, and require business quality, safety, latency, memory, and goodput gates. That process turns a low-bit file into a deployable, auditable model artifact.

References

• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• vLLM Quantization Documentation
• Transformers bitsandbytes Documentation
• GGUF Specification
• NVIDIA Model Quantization Concepts