What is AWQ?

AWQ (Activation-aware Weight Quantization) is a post-training, weight-only quantization method that uses activation distributions to identify salient weight channels and searches equivalent per-channel scaling that reduces their quantization error without retaining a mixed-precision subset.

Quick Facts

Full NameActivation-aware Weight Quantization
Created2023 by Ji Lin et al. (MIT)
SpecificationOfficial Specification

How It Works

AWQ starts from the observation that a small fraction of weight channels can be especially important for model output when viewed through activation magnitudes. It collects activations from a calibration set, searches channel scaling factors, and applies an equivalent transformation before low-bit weight quantization. The published method does not require backpropagation or weight reconstruction. AWQ describes how an artifact is produced; it does not guarantee a specific file size, latency, throughput, or quality level. Those outcomes depend on the base checkpoint, calibration coverage, bit width, group size, packing layout, runtime kernel, hardware, and workload. AWQ and GPTQ therefore require controlled comparison rather than a universal ranking.

Key Characteristics

  • Post-training and weight-only quantization method
  • Uses activation distributions to identify salient weight channels
  • Applies equivalent per-channel scaling to reduce quantization error
  • Does not require backpropagation or weight reconstruction
  • Requires representative calibration data and a pinned recipe
  • Depends on compatible packing layouts, runtime kernels, and hardware

Common Use Cases

  1. Evaluating low-bit weight artifacts for GPU inference
  2. Reducing weight memory traffic when a supported kernel is available
  3. Packaging an LLM for a runtime with verified AWQ compatibility
  4. Comparing weight-only methods under a controlled calibration contract
  5. Increasing serving capacity after business quality and goodput validation

Example

loading...
Loading code...

Frequently Asked Questions

How does AWQ work?

AWQ observes activation distributions from a calibration set to identify salient weight channels. It then searches equivalent per-channel scaling that makes those weights easier to quantize. The corresponding inverse scaling is applied to activations, preserving the original linear operation before rounding error is introduced.

Is AWQ always more accurate than GPTQ?

No. AWQ and GPTQ optimize weight quantization differently, and neither has a universal quality advantage. Results depend on the checkpoint, calibration data, recipe, evaluation tasks, and implementation. Compare artifacts derived from the same baseline with the same tokenizer, chat template, workload, and quality gates.

Does AWQ make inference faster?

Only when the runtime and hardware provide an efficient kernel for the artifact's exact packing layout. Reduced weight traffic can improve performance, but dequantization overhead, operator fallback, small batches, KV-cache pressure, or repacking can erase the gain. Measure TTFT, inter-token latency, throughput, and goodput.

How much calibration data does AWQ need?

There is no universal sample count. The calibration set must represent production languages, domains, prompt lengths, tools, and sensitive task slices. Record the dataset revision, sampling policy, and checksum, then test whether the selected scales remain stable and preserve downstream behavior.

Which runtimes support AWQ?

Runtime compatibility changes by version, model architecture, tensor layout, kernel, and hardware. Consult the target runtime's current official support matrix, pin the version, verify the loaded kernel path, and benchmark the exact artifact. Successful loading alone does not prove an efficient or correct production path.

Related Terms

Quantization

Quantization is the process of mapping model weights, activations, or KV-cache values from a higher-precision numeric representation to a lower-precision representation with explicit scales, zero points, and granularity. It can reduce storage, memory traffic, or compute cost, but it does not guarantee faster inference or acceptable model behavior.

Inference

Inference (model inference) is the execution of a trained model on new input to produce a prediction, score, embedding, classification, or generated output without intentionally updating that model's parameters during the request.

KV Cache

KV Cache (Key-Value Cache) is per-layer inference state that stores the attention Key and Value tensors for retained tokens, allowing autoregressive decoding to reuse them instead of projecting the full prefix again.

LLM

LLM (Large Language Model) is a neural language model trained to predict, score, or generate token sequences from learned parameters and supplied context. Its task behavior depends on the model revision, tokenizer, prompt, decoding policy, tools, retrieval, and evaluation protocol.

Related Articles

LLM Quantization: Methods, Runtime Fit, and Evaluation

Design an LLM quantization workflow across weights, activations, and KV cache. Compare GPTQ, AWQ, SmoothQuant, GGUF, runtime support, and quality gates.

2026-02-21

KV Cache for LLM Inference: Memory, Reuse, and Tuning

Learn how KV cache works in LLM inference, calculate memory per token, compare dynamic, static, paged, offloaded, and quantized caches, and tune prefix reuse.

2026-04-06

AI Accelerator Selection: GPUs, TPUs, and Custom Silicon

Select AI accelerators with a reproducible workload contract. Compare GPUs, TPUs, and custom silicon using memory, latency, goodput, software fit, migration risk, and complete TCO.

2026-05-22