How Much Accuracy Does W4A16 + KV Cache Quantization Lose?

Quantizing weights to W4A16 makes it possible to run Llama-3-70B on a single A100 80GB. But as batch size grows or sequences get longer, the KV cache starts eating into the remaining memory. Dropping the KV cache to FP8 or INT4 frees up room for larger batches — but does accuracy degrade by simply adding the KV cache quantization loss on top of the weight quantization loss, or does something worse happen?

Running the combinations directly reveals that results vary much more than you might expect. W4A16 + KV FP8 keeps the loss at a manageable level, but W4A16 + KV INT4 can cause perplexity to collapse past a threshold on the same model. The reason is that the two error sources don't add up independently.

The Dual Structure of Memory Pressure

Weight memory is static — it's fixed once the model is loaded. Loading Llama-3-70B in BF16 takes roughly 140 GB; quantizing to W4A16 AWQ brings that down to about 37 GB, which fits on a single A100 80GB.

The problem comes next. For Llama-3-70B (80 layers, GQA with 8 KV heads, head dimension 128), the KV cache at batch size 16 and sequence length 8192 works out to:

2(K,V) × 80(layer) × 8(KV head) × 128(dim) × 16(batch) × 8192(seq) × 2bytes(BF16)
= ~40 GB

The 43 GB left after quantization gets filled right back up by the KV cache. Switching to KV FP8 cuts that to 20 GB; KV INT4 cuts it to 10 GB, freeing up that much more room for batches. No matter how aggressively you squeeze weights to INT4, at long contexts and large batch sizes the KV cache reclaims that space — so if you want to push memory utilization to the limit, you have to quantize both.

Where the Errors Collide — The Attention Crossover

Weight quantization error is applied to the Q, K, and V projection matrices. When an INT4-quantized weight matrix is multiplied by the input hidden vector, the projection output Q picks up an error ε_q_w. KV cache quantization error is introduced when K and V are compressed to INT8 or INT4 for storage and then read back — at that point K picks up an error ε_k.

The point where both errors meet is the attention score computation:

score = softmax((Q + εq_w)(K + εk)^T / √d)

Expanding:

(Q + εq_w)(K + εk)^T = QK^T + Q·εk^T + εq_w·K^T + εq_w·εk^T

The last term, εq_w·εk^T, is the cross-error. If each error has magnitude σ, this term is on the order of σ², which is small compared to the individual error terms (order σ). However, the softmax exponential amplifies this nonlinearly, and when outlier channels overlap across both errors, the cross-error can completely distort the attention distribution in specific heads.

The LLM.int8() paper showed that outlier channels in Transformer layers concentrate in a fixed small subset. QServe's SmoothAttention provides indirect empirical evidence that this cross-error is real — scale adjustment alone recovered 0.05 of the 0.14 perplexity increase caused by KV INT4 alone, which implies the cross term is nonzero.

Perplexity Drop by Combination

The numbers below are drawn from the QServe and WKVQuant papers, summarized for Llama-family 7–8B models on WikiText-2 perplexity. Experimental settings differ across papers, so treat these as approximate.

W4A16 + KV FP8 adds roughly 0.1 of KV FP8 error on top of the 0.40 from weight quantization. W4A16 + KV INT4 exceeds a 0.96 drop and swings widely depending on model and dataset. The real problem isn't the point estimate — it's that variance blows up. Under certain conditions, perplexity degrades sharply past a threshold.

KIVI reports small perplexity loss when pushing KV cache to 2 bits while keeping weights in BF16. But that's with BF16 weights. Adding KV INT4 on top of W4A16 changes the cross-error regime entirely, so you can't look at KIVI's results and conclude that W4A16 + KV INT4 is safe.

Which Layers Are More Sensitive

KV cache quantization error is not distributed uniformly across layers. KVQuant and KVSink both point to the same two patterns.

Early layers (roughly layers 0–3) suffer from the attention sink phenomenon, where an outsized attention score concentrates on the first token. When that first token's K vector is quantized to INT4, the error propagates across the entire attention distribution. KVSink reports that keeping sink token K/V in FP16 alone recovers more than 95% of FP16 perplexity.

Late layers are sensitive because there is no opportunity for error correction downstream. Since the layer output feeds directly into the final prediction, a distorted attention pattern in a late layer changes the generated token. Middle layers are comparatively robust — errors have some chance of being partially canceled out by the next layer.

This distribution is the rationale behind mixed-precision KV strategies. KVTuner profiles per-layer sensitivity and assigns FP8 or INT8 to early and late layers while using INT4 for middle layers. Within the same memory budget, this yields a noticeable perplexity improvement over uniform INT4.

Weight quantization error also varies by layer. Since the sensitive regions of both errors are likely to overlap in early and late layers, running a combined quantization scheme warrants keeping the KV cache bit-width higher in those regions.

Practical Selection Criteria

The tradeoffs by combination are as follows:

A PPL drop of 0.7 is a practical threshold. Beyond that, you start seeing 1–2%+ accuracy loss on downstream tasks like MMLU, with a more pronounced impact on code generation and complex reasoning.

There is a sequencing principle here. When you need to reclaim more memory, it's safer to push weight quantization down first rather than reducing the KV cache bit-width first. Dropping weights from W8 to W4 keeps the error confined within the projection matrices. Dropping the KV cache to INT4 spreads cross-errors across the full attention sequence. The pragmatic balance between quality and memory efficiency is to quantize weights first, then stop the KV cache at FP8.

This principle matters more as context length grows. Weight error accumulates proportionally with the number of layers and is independent of sequence length. KV cache error accumulates directly as sequences grow longer, since more error vectors pile up. At 32K context, KV cache error scales proportionally compared to 4K, so in long-sequence settings, keeping KV at FP8 rather than INT4 is the right call.

Enabling This in vLLM and What to Watch Out For

The basic setup for applying weight quantization and KV cache quantization simultaneously in vLLM:

# W4A16 (AWQ) + KV FP8
vllm serve meta-llama/Llama-3-8B \
  --quantization awq \
  --kv-cache-dtype fp8_e4m3

# W4A16 (GPTQ) + KV FP8
vllm serve meta-llama/Llama-3-8B \
  --quantization gptq \
  --kv-cache-dtype fp8_e5m2

The vLLM documentation notes that --kv-cache-dtype-skip-layers can exclude specific layers from FP8 KV cache quantization. Excluding early attention sink layers or sliding window attention layers improves perplexity compared to uniform FP8.

Using the Flash Attention 3 backend together with FP8 KV cache runs the entire attention operation in the FP8 domain. In this mode, Q is additionally quantized to FP8, which adds another error path if weights are already at INT4. The vLLM 2026 blog post reports a cumulative 44% throughput gain from combining linear quantization with KV cache quantization, but calibration scales must be measured for quality-sensitive tasks.

The AWQ + KV INT8 combination was explicitly unsupported in older versions of vLLM. Recent versions have broadened support, but runtime errors can still occur depending on CUDA version and model architecture. If an error message mentions both kv-cache-dtype and quantization, searching GitHub Issues first is the fastest path forward.

SGLang supports the same --quantization and --kv-cache-dtype arguments and automatically recognizes AWQ/GPTQ format checkpoints.