KV Cache Quantization: Why It's Different from Weight Quantization

KV cache quantization is an area where even engineers who understand weight quantization can easily fall into traps. The intuition built from weight quantization — "INT8 cuts memory in half with negligible quality loss" — gets applied directly to KV caches, and engineers only realize something is different after hitting long-context quality degradation.

Why KV Cache Quantization Is a Separate Problem

Weights are frozen after training. Run a calibration dataset through once before converting to INT8, compute the scale factors, and you're done. KV caches are different. Every forward pass generates and accumulates new K and V tensors for each new token, and their distributions shift with the input sequence and position. Even after the model finishes prefill, new tensors are appended at every decode step, so pre-computing a fixed scale factor and reusing it doesn't work in principle.

Weight quantization fixes scales at PTQ time and reuses them throughout serving. KV cache quantization must determine scales at runtime for each request; otherwise dynamic range is lost. This difference narrows the precision options considerably.

The error propagation paths also differ. Weight quantization errors spread as activations pass through layers, but they don't feed back through the attention mechanism. KV cache errors do. Attention scores reference quantized K tensors, and softmax outputs are used to take weighted sums over quantized V tensors, so every prior token's quantization error in the sequence affects the current token's output. The longer the sequence, the longer this error path grows.

Why Key and Value Tensors Have Different Distributions

The KIVI paper (Liu et al., ICML 2024) analyzed KV cache element distributions across Llama, Falcon, and Mistral. The conclusion: Key and Value tensors have fundamentally different distributions.

Key tensors have outliers concentrated along the channel (head_dim) dimension. Certain channels consistently exhibit much larger absolute values than others — these channels have a higher mean magnitude overall. When you compute a single scale across all tokens, those outlier channels consume the precision budget for the rest. This is why per-channel quantization — a separate scale factor per channel — is the right fit for Keys.

Value tensors have relatively uniform distributions. The magnitude difference across channels is much smaller than in Keys; instead, variation occurs along the token dimension. Per-token quantization is more efficient for Values.

Applying the same quantization scheme to both Keys and Values produces disproportionate error on the Key side. This is why KIVI proposes an asymmetric strategy — per-channel INT2 for Keys and per-token INT2 for Values — achieving a 2.6× peak memory reduction at 2 bits while preserving quality. KVQuant extends this further by quantizing per-channel Keys before RoPE is applied, pushing the approach to support inference at context lengths in the millions of tokens.

Context Length and Error Accumulation

Accuracy degradation from KV cache quantization scales with context length. Attention at the current position references K and V from every prior position. With a 4,096-token sequence, you're taking a weighted sum over 4,096 quantized K and V pairs; with a 32K sequence, that's 32,768. Even small per-element errors get amplified by the number of references.

A pattern observed consistently across multiple studies: most LLMs are actually less sensitive to KV cache quantization than to weight quantization at contexts below 4K, but sensitivity reverses at 16K–32K and beyond. RAG workloads regularly exceed 16K context by concatenating multiple retrieved chunks, and running KV cache at INT4 in that regime produces cases where the model fails to correctly reference the retrieved evidence.

The vLLM FP8 KV cache benchmark quantifies this pattern across precision levels. On the MRCR long-context retrieval task at 128K tokens, FP8 recovers 97–98% of BF16 AUC. FP8 holds up this well because the E4M3 format has exponent bits that dynamically adjust the representable range instead of clipping outliers. Fixed-point INT8 incurs large clipping errors when outliers spike. As context grows, this difference accumulates, widening the accuracy gap between FP8 and INT8.

INT8, FP8, INT4: Measured Numbers

Llama-3-70B uses GQA: 80 layers, 8 KV heads, head_dim 128. KV cache per token in FP16:

2(K+V) × 80 layers × 8 KV heads × 128 head_dim × 2 bytes = 327,680 bytes ≈ 0.31 MB/token

Total KV cache (GB) by context length and batch size:

Running Llama-3-70B (roughly 140 GB of FP16 weights) on A100 80GB with tensor parallelism, the 32K × batch 8 configuration alone requires 80 GB just for the FP16 KV cache. Switching to INT8/FP8 drops the KV cache to 40 GB, freeing headroom to increase batch size or extend context.

vLLM FP8 measured results: on Llama-3.1-8B at concurrency 8 with ~20K input tokens, FP8 delivered 14.9% higher throughput and 14.8% lower decode ITL compared to BF16. Usable KV cache capacity doubled from 35,792 to 71,584 tokens under the same conditions. That said, per SqueezeBits' comparison, enabling FP8 KV cache in vLLM disables the FlashAttention-2 backend, which can actually reduce throughput in some scenarios. TensorRT-LLM supports both INT8 and FP8 and handles FP8 attention operations with a more complete internal implementation. MMLU accuracy was essentially unchanged from FP16 with 8-bit KV cache on both frameworks (~0.679).

Configuration Examples

vLLM (FP8 KV cache, CUDA 11.8+):

vllm serve meta-llama/Llama-3-70b-Instruct \
  --kv-cache-dtype fp8_e4m3 \
  --tensor-parallel-size 4

The default scale is 1.0. For better accuracy, run calibration with llm-compressor beforehand and pass the resulting scale JSON. To exclude specific layers from quantization, use flags like --kv-cache-dtype-skip-layers sliding_window.

TensorRT-LLM (INT8 KV cache):

trtllm-build \
  --checkpoint_dir ./llama-3-70b-hf \
  --int8_kv_cache \
  --tp_size 4 \
  --output_dir ./engines/

Scale factors are computed automatically from calibration data at build time. Switch to FP8 with the --fp8_kv_cache flag; when used alongside FP8 model weights, the entire attention operation runs in FP8, which yields greater gains.

When to Enable KV Cache Quantization

Start by diagnosing where the memory pressure is coming from. Monitor GPU memory at batch size 1 with short context — that gives you the baseline memory consumed by weights. The increase as you extend context is the KV cache:

# Estimate KV cache memory per token (FP16)
kv_per_token_bytes = 2 * num_layers * num_kv_heads * head_dim * 2
kv_cache_gb = (kv_per_token_bytes * seq_len * batch_size) / (1024**3)

KV cache quantization makes sense when: context is 8K or longer and you need larger batch sizes for serving throughput, or weights are already compressed as much as practical and the KV cache is the bottleneck. In these cases, start with FP8 — its wider dynamic range makes it safer for long contexts than INT8 — and keep it if accuracy holds.

Situations to avoid it: short conversational serving with context under 4K (the savings are small and the overhead isn't worth it); considering INT4 for precision-sensitive long-context inference (errors are larger than expected). In environments where vLLM's prefix caching is active, the higher the cache hit rate, the more FP8 quantization's capacity gains matter.

Scale factor reuse trap: reusing prefill-computed scale factors throughout decoding means the tensor's dynamic range shifts as new tokens are appended, diverging from what was seen at prefill. The reason vLLM's default scale of 1.0 holds up reasonably well with FP8 is that E4M3's wide representable range absorbs scale errors — the same approach does not work with fixed-point INT8. If you choose INT8, use a configuration with per-token dynamic scaling.