Why GQA's KV Cache Savings Don't Fully Reduce TPOT

GQA reduces KV cache — that much is established. It's also well known that Llama-3 70B uses one-eighth the KV cache of Llama-2 70B. But in actual serving workloads, TPOT doesn't drop by 8×. In many cases it doesn't even drop by 2×. To understand why, you need to look at where the GPU actually spends its time during the decode phase.

MLA (Multi-head Latent Attention) complicates this picture further. As DeepSeek-family models make their way into production, the claim that "MLA is better than GQA" is circulating — but without understanding under which conditions it holds and under which it doesn't, you can't use either effectively.

KV Cache Size by Attention Variant: Starting with the Math

MHA, MQA, GQA, and MLA all ultimately aim to reduce KV cache bytes per token per layer. The formula is the same across all of them; only n_kv differs.

KV cache bytes = n_layers × n_kv × d_head × 2(K+V) × L × B × bytes_per_elem

Using the Llama-3 70B architecture (n_layers=80, n_heads=64, d_head=128, FP16) with a context of 8K tokens per sequence, the differences between variants are stark.

At batch size 32, MHA requires 688 GB and GQA-8 requires 86 GB. On an 8× A100 80GB server, MHA at batch 32 runs out of VRAM; GQA-8 fits on 2 cards. This capacity gap directly determines throughput.

MLA uses a different formulation entirely and is covered in its own section below.

Why TPOT Doesn't Drop Along with KV Cache

The decode phase is memory-bound. Execution time is dominated by the GPU waiting for data to arrive from HBM. So if KV cache bytes decrease, HBM loads should decrease, and TPOT should follow — but that reasoning has a missing term.

At each decode step, the GPU isn't only reading KV cache from HBM. It re-reads the weight matrices on every token. Loading Llama-3 70B in BF16 means 140 GB of weights alone. With H100 SXM's HBM3 bandwidth at approximately 3.35 TB/s, reading those 140 GB takes 41 ms in theory. This cost is identical whether you use GQA or MHA.

KV cache loading cost scales with sequence length and batch size. At batch=1 with 8K context, GQA-8's KV cache is 2.7 GB — small relative to 140 GB of weights. As batch size grows, this reverses. At batch=64 with 8K context, GQA-8's KV cache reaches 172 GB, exceeding the weight footprint. Beyond this crossover point, KV cache savings start showing up meaningfully in TPOT.

From a roofline perspective: the FP16 ridge point is roughly 153 FLOPs/byte on A100 and roughly 295 FLOPs/byte on H100. Decode arithmetic intensity at batch=1 is often below 10 FLOPs/byte — well under 1/15 of the ridge point. In this regime, any reduction in HBM loading only touches part of the bottleneck.

There are two paths through which KV cache reduction translates to TPOT improvement. The first is a direct reduction in KV loading bytes, which is only significant at large batch + long context. The second is fitting more sequences into the same VRAM, which allows increasing batch size itself — and this path drives throughput gains even in the small-batch regime. From a serving throughput standpoint, the second path is the one that fires more often.

What Makes MLA Different: Serving Implications of Low-Rank Compression

DeepSeek-V2 (arXiv:2405.04434) introduces MLA, which replaces per-head high-dimensional KV tensors in the cache with a single low-rank latent vector. At attention computation time, the K and V for each Q head are recovered via up-projection.

Using DeepSeek-V2's numbers: n_h=128 query heads, d_h=128 head dimension, KV compression dimension d_c=512, decoupled RoPE dimension d_h^R=64. Comparing cache size per token per layer against MHA:

That's roughly a 57× difference. The paper frames this as equivalent to approximately 2.25 GQA KV heads (576 / (2 × d_h) = 576/256 ≈ 2.25). Across DeepSeek-V2's 60 layers with a single sequence at 8K context, where MHA-equivalent would consume roughly 30 GB, MLA uses 540 MB. DeepSeek-V2 reports a 93.3% KV cache reduction relative to its predecessor, DeepSeek 67B.

The structural difference from GQA matters here. GQA reduces the number of KV heads, cutting both cache size and attention FLOPs together. MLA stores compressed latent vectors in the cache but must reconstruct full K and V for each Q head via up-projection at every decode step. Loading bytes decrease; FLOPs increase — arithmetic intensity goes up. On H100, where compute capacity relative to HBM bandwidth is high, the bytes reduction wins. On older GPUs with relatively wider bandwidth and lower compute ceilings, the added FLOPs can become a liability.

vLLM has supported DeepSeek MLA natively since v0.7.1 via "matrix absorption," which mathematically fuses the up-projection with the Q-side down-projection so that decode-time attention requires no explicit reconstruction step. This keeps the latent cached while minimizing FLOPs overhead. The exception is the decoupled RoPE component (d_h^R=64): because it carries position dependence, absorption doesn't apply, so it is stored and handled explicitly — which is why the cache size is 576 rather than 512.

Behavioral Characteristics by Batch Size × Context Length

The relative magnitude of weight-loading cost versus KV cache-loading cost determines which architecture wins. Summarized:

In the small-batch, short-context regime, it's hard to claim GQA-8 meaningfully beats MHA on TPOT. Weight loading is so dominant that KV-side savings often fall within measurement noise. In the large-batch, long-context regime, GQA consistently handling more sequences than MHA at the same latency is well-established.

MLA dominates GQA on cache capacity across all regimes. On TPOT, on H100 and above, it is on par with or slightly better than GQA. There are reports of MLA being slower than GQA in the short-context, small-batch regime due to up-projection overhead, so "MLA is always faster" is not a safe claim.

When running GQA models in vLLM, incrementally increasing --max-num-seqs and observing TPOT shows a characteristic pattern: TPOT is nearly flat at low batch sizes, then as KV memory saturates, GQA's capacity advantage converts into throughput gains. TTFT is a different story — since prefill is compute-bound and attention FLOPs do decrease with GQA, a modest GQA > MHA advantage on TTFT appears regardless of context length.

Where Quality Degradation Shows Up

The GQA paper (Ainslie et al., 2023) shows that MQA exhibits quality degradation relative to MHA, while GQA occupies a middle ground. The degradation is most visible on long-context recall tasks.

On Needle-in-Haystack benchmarks, as context extends to 16K or 32K tokens, MQA recall accuracy collapses sharply at certain depth ranges. GQA lets you tune this degradation curve via group count — more groups moves you toward MHA quality, but reduces cache savings. Llama-3 70B's choice of GQA-8 over GQA-2 is unlikely to be arbitrary. GQA-2 probably produced unacceptable quality loss on internal long-context benchmarks.

MLA has its own failure mode: numerical error accumulation in the compression-reconstruction path. At very long contexts, the interaction between decoupled RoPE and latent reconstruction can perturb attention score distributions. When DeepSeek-V2 claims to "exceed MHA," that reflects a model specifically trained with d_c and the training procedure tuned for this architecture — applying the same compression ratio to a different architecture may not produce the same result.

At this point, MLA adoption is concentrated in the DeepSeek family, so independent third-party long-context benchmarks haven't accumulated in volume. The limited number of direct GQA-8 vs. MLA comparisons on LongBench or Needle-in-Haystack at matched parameter counts is what makes confident conclusions difficult right now.

When the Serving Engineer Doesn't Get to Choose the Model Architecture

Model architecture is usually already fixed by the time serving engineers are involved. There are three levers left to adjust.

TP degree and KV head count alignment. When tensor parallel degree exceeds n_kv_heads in a GQA model, KV heads get replicated. Running Llama-3 70B (n_kv=8) at TP=4 gives 2 KV heads per GPU — fine. At TP=16, vLLM replicates KV heads to handle the mismatch. Replicated KV cache is stored redundantly, which erodes GQA's memory advantage. Satisfying num_kv_heads % tensor_parallel_size == 0 is the safe path; when you can't, be explicit about the replication overhead you're accepting.

KV cache quantization. Passing --kv-cache-dtype fp8 in vLLM drops GQA-8's 2.7 GB/seq to 1.35 GB/seq. GQA quantizes naturally at the KV head granularity. With MLA, quantizing the latent vector to FP8 can amplify error after up-projection, so more care is warranted. Whether this actually causes problems depends on the model and task — Needle-in-Haystack is the most direct way to verify.

Prefix cache hit rate. Smaller KV cache entries mean more prefix blocks fit in the same cache memory. GQA uses prefix cache more efficiently than MHA; MLA's much smaller latent blocks raise the theoretical hit rate further. That said, verify how vLLM manages MLA latent cache blocks — if block size configuration is off, hit rates won't reach the theoretical ceiling.

None of these three levers is as impactful as choosing a different attention architecture to begin with. Even so, a wrong TP degree choice alone can throw away half the benefit of using GQA, which is why it's the first thing to check.