Why Keys Are More Sensitive Than Values in INT8 KV Cache Quantization

When you apply INT8 quantization to the KV cache and monitor quality, you run into a strange pattern: Value quantization causes less degradation than expected, while only the Key side causes attention scores to collapse in certain layers. Same 8 bits, so why this asymmetry?

The answer lies in the structure of the attention computation. Keys and Values may look similar as tensors, but their computational paths are fundamentally different — and that difference determines how quantization error propagates.

Computational Paths for Key and Value

The attention output in equation form:

Score = softmax(QK^T / sqrt(d_k))
Output = Score · V

Key quantization error first enters at the QK^T dot product, where scores are formed. The problem comes next: softmax. Because softmax is a nonlinear function, even small perturbations in the score values can distort the entire probability distribution. If the raw score difference between two tokens shifts from 0.5 to 0.9, the resulting attention weight ratio after softmax changes by a much larger factor. This amplification is what makes Key quantization error dangerous.

Value quantization error takes a different path. Score · V is a simple linear weighted sum — a weighted average of V by scores that have already been normalized through softmax. A small error in V propagates linearly and predictably. If V is slightly off while the score distribution is intact, the output is off by the same proportion. No nonlinear amplification.

KIVI (Liu et al., ICML 2024) empirically confirmed this asymmetry. By applying per-channel quantization to the Key cache and per-token quantization to the Value cache — an asymmetric strategy — they compressed down to 2 bits while preserving quality on Llama, Falcon, and Mistral models. Peak memory dropped 2.6×, and throughput improved 2.35–3.47×.

Outlier Distribution in Key Channels

The reason per-tensor INT8 quantization breaks down especially badly on Keys is channel-wise variance asymmetry.

Slice the Key tensor along the channel dimension, and you'll commonly find a handful of channels with value distributions more than 10× larger than the rest. Analyses of KV caches in Llama-family models repeatedly show a pattern where a small number of channels account for a disproportionate share of the total variance. Per-tensor quantization uses a single scale for the entire tensor, so that scale is determined by the range of the outlier channels. As a result, normal channels with no outliers occupy only a tiny fraction of the available INT8 representation range — most of the quantization resolution is wasted.

The Value tensor has less of this problem. Channel-wise variance is relatively uniform, so a single per-tensor scale covers the full tensor reasonably well. KVQuant (Hooper et al., NeurIPS 2024) explicitly notes that "the Key matrix has distinct outlier channels with larger average magnitudes than other channels," and explains that per-channel Key quantization is designed to prevent those outlier channels from contaminating the rest.

Per-channel quantization assigns an independent scale/zero-point per channel, so even if an outlier channel spikes, its scale applies only to that channel. Channels without outliers get their own scale matched to their distribution, making full use of the INT8 representation range.

INT8 per-tensor / per-channel / FP8 e4m3: Empirical Comparison

KVQuant's evaluation of 3-bit quantization on LLaMA-7B with Wikitext-2 shows a clear gap between approaches:

Ablations comparing Key-only vs. Value-only quantization consistently show smaller degradation on the Value side. This gap is especially pronounced on the GSM8k math reasoning task. The longer the reasoning chain, the more the effects of mangled Key outlier channels accumulate.

FP8 approaches this problem from a different angle. The e4m3 format uses a 4-bit exponent and 3-bit mantissa, with a narrow representable range of ±240 but higher mantissa resolution — giving better precision for small values than INT8 per-tensor. Combined with per-channel calibration scales, it handles outlier channels comparably to per-channel INT8. The e5m2 format assigns 5 bits to the exponent, giving wider range at the cost of precision. If the Key outlier channels have a very wide value range, e5m2 is preferable; if the range is narrow and precision is the priority, e4m3 is the better choice.

Layer Depth and Sensitivity

Tolerance to KV quantization error varies by layer.

Early-layer attention handles local patterns like token position and syntactic structure. If Key outlier channels get mangled in these layers, the model loses its grip on basic sequence structure, and that error propagates through all subsequent layers. Deeper layers then have to reconstruct semantics from an already corrupted representation — which rarely succeeds. Conversely, deep layers are relatively tolerant of quantization error as long as the representations coming in from earlier layers are healthy.

For this structural reason, applying the same quantization scheme uniformly across all layers is inefficient. That's why follow-up work like KVTuner (2025) moves toward mixed-precision KV quantization, measuring per-layer sensitivity and assigning bit widths accordingly.

vLLM exposes the --kv-cache-dtype-skip-layers flag to exclude specific layer indices or layer types from FP8:

# Keep early layers and specific indices in BF16
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --kv-cache-dtype fp8 \
  --kv-cache-dtype-skip-layers 0 1 23

# Exclude all sliding-window layer types
vllm serve <model> --kv-cache-dtype fp8 \
  --kv-cache-dtype-skip-layers sliding_window

This option is particularly effective for hybrid models that use sliding-window attention. Sliding-window layers have short KV lengths, so the memory benefit from quantization is small while the quality sensitivity remains high.

Practical Recommendations

Current major serving frameworks don't expose an official interface for applying different quantization schemes to Keys and Values separately. Both vLLM and SGLang apply the same dtype to both via --kv-cache-dtype. Reproducing the asymmetric strategy demonstrated in KIVI and KVQuant requires a separate patch or a custom attention kernel.

The realistic options are these three:

For long-context summarization or complex reasoning tasks, the safe approach is to start with fp8_e4m3 + early-layer skip and measure task-specific metrics. For short classification or simple Q&A, plain fp8 is unlikely to produce a perceptible quality difference in production.

Using the default scale=1.0 without calibration is worse than running dataset-based calibration with llm-compressor to obtain calibration scales — the latter preserves accuracy on models where Key channel outliers are large. The vLLM official documentation lists this calibration path as the recommended approach.

It's unclear when Key/Value split configuration will be officially supported at the framework level. The KIVI implementation is available publicly and KVQuant code is also out there, but vLLM mainline currently has no official option for simultaneous Key per-channel + Value per-token quantization. As demand for long-context serving grows, pressure to implement this will increase.