Why KV Cache Offloading Is Slower Than You Expect

GPU memory runs out, and the first thing you reach for is cpu_offload_gb or --n-gpu-layers. One config line and you're past the OOM. The problem is the distance between "it runs" and "it's usable."

The real bottleneck in KV cache offloading is the repeated PCIe and DRAM round-trip on every decode step. This cost scales worse with batch size than with context length.

Memory Hierarchy Bandwidth: The Numbers First

Knowing which path your data travels makes offloading costs immediately clear.

For Llama-3-8B (32 layers, GQA with 8 KV heads, head dim 128, bf16), the KV size per layer is:

K or V (seq 128K, 1 layer)
  = 128,000 tokens × 8 KV-heads × 128 head-dim × 2 bytes
  ≈ 250 MB

K + V combined ≈ 500 MB / layer

Wall-clock transfer time for that 500 MB across each tier:

Every decode step must pass through all 32 layers on this path. Storing the full KV in CPU DRAM means 32 × 15.6 ms = ~500 ms per generated token spent purely on PCIe transfers. In practice, some overlap with compute is possible — but when a single layer's attention computation finishes in 0.25 ms on HBM while the transfer takes 62× longer, there is nothing left to hide behind overlap.

When Transfers Happen Inside the Decode Loop

Prefill runs once. Decode repeats every token. That's where the offloading trap lies.

At each decode step, processing layer i requires loading the full KV cache for layer i from the offload location back to GPU. Once attention completes, the new token's KV is written back out. This repeats sequentially across every layer.

With CUDA streams and PCIe DMA, you can overlap the layer i+1 prefetch with layer i computation. FlexGen (Sheng et al., ICML 2023) implements double buffering this way — six logical threads simultaneously handle the current layer's compute, the next layer's KV prefetch, and the previous layer's writeback.

Overlap only works when compute time ≥ transfer time. At seq 128K, one layer's attention computation takes roughly 0.25 ms (measured by HBM read time for 500 MB). PCIe 4.0 takes 15.6 ms. The GPU finishes compute and then stalls for 15+ ms waiting for the next block.

Increasing batch size looks like it helps. At batch=4, compute volume quadruples, seemingly leaving more room to hide the transfer. But batch=4 also means managing KV for 4 concurrent requests. The total KV transfer volume grows at the same rate as the batch size. This is why TPOT degrades in steps rather than linearly — the moment the batch crosses PCIe saturation threshold, every request enters the stall window simultaneously.

llama.cpp vs. vLLM vs. FlexGen: What Gets Moved Differs

The same word "offload" covers fundamentally different mechanisms.

llama.cpp (--n-gpu-layers)

Splits layers between GPU and CPU. --n-gpu-layers 20 means the top 20 layers run on GPU; the rest run on CPU. This isn't just moving KV — the full layer weights and computation itself move to CPU. The bottleneck is CPU compute speed rather than PCIe transfer. There's also synchronization overhead at every layer boundary to hand activations between GPU and CPU.

This structure is clearly visible in real measurements on an RTX 3070 Ti (8 GB VRAM) with Llama-3-8B Q4: CPU-only runs at ~9–10 tok/s, GPU-only at ~34 tok/s. Splitting half the layers onto GPU doesn't yield the naïve midpoint of 22 tok/s — it tops out at 20–24 tok/s, because synchronization overhead between GPU and CPU compute prevents linear interpolation.

# Example: offload 20 of 35 layers to GPU (Llama-3-8B)
./llama-cli -m llama-3-8b.Q4_K_M.gguf \
    --n-gpu-layers 20 \
    -p "summarize this long document: ..." \
    -n 512

vLLM (cpu_offload_gb / kv_offloading_size)

KV cache only. Weights stay on GPU; only KV blocks are moved to CPU DRAM. The official docs describe this as "virtually expanding GPU memory" — --cpu-offload-gb 10 makes a 24 GB GPU behave like a 34 GB GPU. In practice, KV blocks are evicted to CPU in LRU order under GPU memory pressure and fetched back on demand. The fact that the official docs include no latency penalty numbers is telling — the feature is positioned as a capacity expansion mechanism.

Starting with vLLM v0.8.3, a kv_offloading_size parameter was added. It specifies the KV buffer size across all ranks in a tensor-parallel setup, and kv_offloading_backend lets you select LMCache instead of the native implementation.

FlexGen and InfLLM

FlexGen manages weights, KV, and activations across GPU/CPU/NVMe, with LP-based automatic scheduling for batch optimization. On a single NVIDIA T4 (16 GB), it ran OPT-175B at 0.69 tok/s without compression and 1.12 tok/s with 4-bit compression. When using NVMe, it enforces sequential access patterns — measured throughput in the experimental setup came in at ~2 GB/s sequential read, well below the theoretical 7 GB/s.

InfLLM (Xiao et al., NeurIPS 2024) uses selective loading. KV is grouped into 128-token blocks, with only 4 representative tokens per block kept on GPU. During attention, only the top-k blocks are loaded from CPU, selected by scoring against the current query's representative tokens. On Mistral-7B over ∞-Bench (average sequence length 145K tokens), InfLLM achieves 57.7% vs. StreamingLLM's 21.5%, while reducing memory and wall-clock time by 34% each compared to full attention.

When Offloading Is Actually Practical

CPU DRAM offloading and NVMe offloading aren't just different points on a bandwidth scale. NVMe delivers 7 GB/s on large sequential IO, but drops below 10% of that for small random IO. A poorly designed block size can make CPU DRAM memcpy faster than NVMe — which is exactly why FlexGen enforces sequential scheduling.

Computing the context length ceiling for TPOT < 50 ms on PCIe 4.0:

Per-layer transfer time = seq_len × (8 heads × 128 dim × 4 bytes) / 32 GB/s
                        = seq_len × 4.096 μs
32-layer total          = seq_len × 131 μs

seq 8K   → ~33 ms for 32 layers  (right at the TPOT 50 ms boundary)
seq 32K  → ~130 ms  (unacceptable)
seq 128K → ~524 ms  (unacceptable)

For throughput-oriented serving, reducing HBM demand through quantization is almost always better than offloading.

Alternatives That Achieve the Same Goal Without Offloading

Applying W4A16 quantization to Llama-3-8B brings weight memory down from 16 GB to 4–5 GB, freeing space for KV. Adding INT8 KV quantization cuts KV memory in half on top of that. This is the most direct way to free memory with no latency penalty.

If you don't strictly need long context, using RAG to inject only relevant chunks and capping the window at 8–16K beats offloading on both latency and memory.

The remaining question is whether PCIe 5.0 becoming mainstream changes this picture. At seq 128K, CPU offload transfer time drops from 524 ms to 262 ms — that's real. But it's still 31× slower than HBM. As long as the bottleneck comes from the layer-sequential structure of the decode loop rather than raw bandwidth, doubling the wire width only solves half the problem.