Where Quantization Loses Accuracy: W4A16, W8A8, and FP8 Compared

Where Quantization Loses Accuracy: A Practical Comparison of W4A16, W8A8, and FP8

If you've ever applied quantization to cut serving costs and found that perplexity barely moves while actual response quality noticeably degrades — especially on tasks requiring precision like coding or math reasoning — you're not alone. This post explains why that happens: which layers and which operations are the source of the loss. It also compares W4A16, W8A8, FP8, GPTQ, and AWQ with real measured numbers.

Compressing Only Weights (W) vs. Compressing Both Weights and Activations (W+A)

LLM inference has two bottlenecks: GPU memory capacity and compute throughput. Which bottleneck a quantization scheme targets determines what kind of technique it is, and conflating the two leads to bad decisions.

W4A16 compresses only the weights (W) to 4 bits. Activations (A) remain at 16 bits, and the actual matrix multiplication runs on FP16 after dequantizing the weights. For Llama-3 8B, BF16 occupies roughly 16 GB of VRAM; W4A16 brings that down to about 6 GB. The memory savings are real. But compute itself doesn't get faster, because the GEMM is still FP16. The benefit is clear when memory bandwidth is the bottleneck (batch size 1–4), but at batch size 32 and above — where compute is saturated — the dequantize overhead can actually hurt.

W8A8 processes both weights and activations in 8 bits, and the matrix multiplication itself runs as INT8 GEMM. On hardware with INT8 Tensor Core support, this genuinely increases compute throughput relative to BF16. Memory footprint is about 8 GB, roughly half of BF16. That said, whether INT8 GEMM is actually faster depends on the hardware and matrix dimensions — more on this later.

FP8 is the floating-point counterpart of W8A8. NVIDIA introduced the E4M3 (4-bit exponent, 3-bit mantissa) and E5M2 (5-bit exponent, 2-bit mantissa) formats, with native FP8 Tensor Core support starting from H100 (Hopper) and Ada Lovelace (RTX 4090 family). Unlike integer INT8, FP8 can represent outlier values as-is across a wide dynamic range, so accuracy loss is substantially smaller than with INT8.

Putting all three on the same tradeoff curve leads to the wrong conclusions. W4A16 is a tool for relieving memory bottlenecks; W8A8/FP8 are tools for increasing compute throughput.

The Loss Is Not Uniform: Per-Layer Sensitivity

If quantization error were spread evenly across all layers, perplexity would be an adequate measure. In practice, the error concentrates in a few specific places.

The most vulnerable are channels with highly concentrated outlier activations. Dettmers et al. (2022) observed that in LLMs with 6B parameters or more, certain hidden dimensions develop activations with extremely large magnitudes — emergent outliers. Clamping these channels into INT8 destroys the information in those channels entirely. Even if only 5–6 out of 128 channels are corrupted this way, downstream task performance drops significantly. This is exactly why perplexity might tick up only slightly while benchmarks like GSM8K or HumanEval take a much larger hit — perplexity underestimates the real loss precisely in these cases.

The Q/K projections in attention are also vulnerable. These are the core layers responsible for positional information and inter-token similarity. Errors here distort attention patterns in long contexts. If you see quality that holds up on short generation tasks but suddenly degrades on long-form reasoning or multi-turn conversations, checking quantization error in the Q/K projections is a good first step.

The first and last layers are also more sensitive than the middle layers, because their value distributions differ enough that the same quantization settings produce relatively larger errors.

LLM.int8() sidesteps this problem with mixed-precision decomposition. Outlier channels exceeding a defined threshold are handled with FP16 matrix multiplication, while the remaining 99.9%+ of values run as INT8 GEMM. Accuracy is preserved, but the FP16 execution path introduces latency overhead — particularly noticeable on smaller models (7–8B).

AWQ takes a different approach to the same problem. It identifies important channels by looking at activation magnitudes, then adjusts the scale of those channels offline using a mathematically equivalent transformation. This is covered in more detail in the GPTQ vs. AWQ section below.

Per-Method Numbers: Llama-3 8B Baseline

The table below summarizes key metrics for each format using Llama-3 8B as the reference model. Perplexity is measured on WikiText-2.

FP8 maintains near-BF16 perplexity because of the dynamic range that floating-point representation provides. Rather than clamping to an integer grid, the exponent bits preserve outlier values. A large-scale evaluation by Red Hat on Llama 3.1 8B/70B/405B confirmed that FP8 recovers over 99% of the original score on the OpenLLM Leaderboard average.

The wide perplexity range for W4A16 (GPTQ/AWQ) reflects how much group size and calibration data affect quality. Group size 128 is the typical setting, and most configurations land around 6.5 in that case.

On throughput, serving Mistral 7B with FP8 on an H100 shows approximately 33% higher output tokens/s and 8.5% lower TTFT compared to FP16, with VRAM dropping from 16 GB to 7 GB. Given this level of accuracy preservation and speed improvement, FP8 is effectively the default choice when H100 hardware is available.

GPTQ vs. AWQ: Why the Same W4 Produces Different Quality

GPTQ approximates the Hessian matrix to correct quantization error. When quantizing each weight to 4 bits, it propagates the error to the remaining weights so that the total error is minimized at the layer level. It's efficient enough to quantize a 175B-parameter model on an A100 in about four hours, and was the W4 state of the art when introduced.

The problem is that the Hessian computation is tied to the activation distribution of the calibration data. Calibrating on general text like C4 or WikiText works well for general inference. But when serving a specialized domain — code, math, medical, etc. — the mismatch between the calibration distribution and the serving distribution skews the direction of error correction. Calibrating on math text improves GSM8K performance, while the same model can drop on HumanEval relative to the default calibration. The optimization meant to reduce error ends up hurting performance in other domains.

AWQ identifies important channels by activation magnitude, without using Hessians. It adjusts the scale of those channels offline via a mathematically equivalent transformation, so that the effective precision of important channels is preserved after quantization. With no backpropagation involved, there's no structural risk of overfitting to the calibration domain. In practice, AWQ tends to score 1–2% higher than GPTQ across benchmarks, and it integrates more cleanly with the INT4 GEMM kernels in vLLM and SGLang.

When calibration data is carefully matched to the serving domain, GPTQ can outperform AWQ on specific tasks. But for the typical use pattern — calibrating on C4 and serving across diverse domains — AWQ is the safer choice.

Hardware Support Determines Real-World Speed

The same W8A8 configuration can be fast or slow depending on the hardware.

H100 (Hopper) natively supports both FP8 Tensor Cores and INT8 GEMM. FP8 W8A8 can theoretically use 2× the FLOPS of BF16, and measured throughput improvements of 33% or more are real.

A100 is different. It supports INT8 GEMM (via cuBLAS) but has no FP8 Tensor Cores. Running an FP8 model on A100 causes vLLM to upcast internally to BF16 before computation, so there's no speed benefit from FP8 — and the casting overhead can actually make it slower. Running W8A8 INT8 on A100 also requires care: cuBLAS INT8 GEMM is not optimized for all matrix sizes and batch configurations, and can be slower than FP16 GEMM in some cases. This is why W4A16 (AWQ) can outperform W8A8 in throughput on A100 — the memory bandwidth savings outweigh the dequantize overhead.

RTX 4090 (Ada Lovelace) technically supports FP8 Tensor Cores. However, with only 24 GB of VRAM, Llama-3 70B won't fit on a single GPU even in FP8 (~35 GB). An 8B model fits comfortably in FP8 (~7 GB), so FP8 is viable for serving 8B models on a 4090. The RTX 3090 (Ampere) doesn't support FP8, making W4A16 the practical best option.

Choosing a Quantization Method

Here's a decision guide based on GPU generation and batch size.

Using W4A16 for compute-bound serving can be slower than BF16. A dequantize step is added before the GEMM, and the GEMM itself is still FP16. The "always use INT4" rule only holds when memory bandwidth is the bottleneck.

Longer contexts (16K+) sustain the memory bottleneck longer due to KV cache pressure, extending the range where W4A16 is beneficial. For high-throughput serving with short contexts and large batches, W8A8 or FP8 is the right choice.

If you're in an environment where FP8 isn't available and need to use W4A16, default to AWQ with a domain-neutral calibration corpus like C4 or Pile-eval. If the serving domain is specialized — code or math — build a separate calibration dataset from samples in that domain. When choosing between GPTQ and AWQ, AWQ is the safer default because it's more robust to calibration domain mismatch.