KV Cache Transfer Costs: Disaggregated Prefill Across NVLink, RDMA, and PCIe

Once the prefill node finishes computation, it must transfer the KV cache to the decode node. How long that transfer takes determines whether PD disaggregation is worth it. On NVLink, a single 4K-context transfer adds 1.4 ms; on 100GbE Ethernet, it's 102 ms — the same architecture, but infrastructure alone drives a 73× difference in added TTFT.

What Determines KV Transfer Volume

KV cache size is the product of the number of layers, KV heads, head dimension, and context length.

KV_bytes = num_layers × 2 × num_kv_heads × head_dim × sizeof(dtype) × seq_len

For LLaMA-2 70B (80 layers, GQA with 8 KV heads, head dim 128, FP16):

= 80 × 2 × 8 × 128 × 2 × seq_len
= 320 KB/token × seq_len

Transfer volume by context length:

GQA alone produces an 8× difference in transfer volume. Modern models adopt GQA not just to save KV memory — in a PD disaggregated setup, it also cuts transfer cost by 8×.

Transfer Time by Interconnect Topology

NVLink is only available intra-node on H100s — these numbers apply when the prefill and decode nodes sit within the same NVSwitch fabric. Once the nodes are physically separated, you're down to InfiniBand or Ethernet.

DistServe (OSDI 2024) measured a single-request KV size of roughly 1.13 GB for OPT-66B at 512 tokens. At an average load of 10 req/s, sustaining that throughput requires 11.3 GB/s — about 90 Gbps — of transfer bandwidth, enough to saturate a single 100 Gbps Ethernet link with KV traffic alone.

Splitwise (ISCA 2024) reported that in clusters equipped with InfiniBand, KV transfer accounted for less than 0.1% of total latency, with 95% of requests seeing transfer delays under 30 ms. That figure assumes the right infrastructure is in place.

Break-Even Point: When Does PD Disaggregation Backfire?

The net gain from PD disaggregation simplifies to:

net gain = reduction in prefill computation time - KV transfer latency

If the KV transfer latency exceeds the time saved compared to running prefill monolithically, disaggregation makes things worse.

Ethernet + 1K context is the worst combination. The transfer alone takes 26 ms, while prefill computation for 1K tokens is on the order of a few milliseconds. There's almost nothing to save from disaggregation, yet you're adding a fixed 26 ms penalty. At small batch sizes, GPU utilization conflicts are infrequent anyway, so the benefit of disaggregation shrinks further.

Ethernet + 32K context is also problematic. Transfer alone takes 819 ms. Unless prefill computation runs into multiple seconds, the transfer cost will exceed any gain from disaggregation. For IB NDR + 8K context, transfer time is around 51 ms. Disaggregation makes sense if prefill exceeds 100 ms, but below that threshold the transfer cost eats into any benefit.

NVLink intra-node resolves this most cleanly — 11 ms even at 32K context, well below typical prefill computation time, making break-even analysis almost unnecessary. However, this configuration requires prefill and decode instances to share the same NVSwitch fabric, which means they can't scale independently the way a fully disaggregated serving setup would. That's a real trade-off in operational flexibility.

Mooncake (Moonshot AI, 2024) works around this problem with a distributed KV cache that spans CPU memory, DRAM, SSD, and NICs. Rather than treating the transfer as a bottleneck, it pre-caches KV data close to where it's needed, and reports up to a 498% improvement in effective request throughput on the Kimi service.

Techniques to Reduce KV Transfer Cost

KV Quantization (FP8 / INT8)

Switching from FP16 to FP8 cuts transfer volume in half. For a 4K-context transfer over 100GbE, that brings latency from 102 ms down to 51 ms. The consensus today is that FP8 is safe for virtually all tasks. INT8 is also fine for most models. INT4 is where model-to-model variance becomes significant — Qwen2.5-7B sees a sharp quality drop at INT4, while Mistral-7B is comparatively stable.

In vLLM, a single flag applies it:

--kv-cache-dtype fp8

SpectrumKV uses mixed precision — FP16 for attention sink tokens and INT4 for lower-importance tokens — keeping perplexity degradation within +2% at 50% transfer reduction. For context, the competing approach PDTrim showed +25% perplexity degradation under the same conditions.

Asynchronous Pipelining

This is precisely why NVIDIA NIXL is designed as an asynchronous point-to-point transfer library. While KV data is in flight, the decode side can begin processing layers that have already arrived, overlapping part of the transfer wait with first-token generation. This is an effective way to reduce real TTFT on Ethernet.

Selective Layer Transfer

Research is ongoing into transferring only the layers that matter most for output quality, rather than all 80. This isn't production-ready yet, but it points toward a path for further reducing PD disaggregation transfer costs.

Measuring It Yourself with vLLM NIXL

You can stand up PD disaggregation locally using vLLM NixlConnector.

Prefill worker:

export UCX_TLS=all
export UCX_NET_DEVICES=all

CUDA_VISIBLE_DEVICES=0 \
VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
vllm serve <MODEL> --port 8100 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}'

Decode worker:

CUDA_VISIBLE_DEVICES=1 \
VLLM_NIXL_SIDE_CHANNEL_PORT=5601 \
vllm serve <MODEL> --port 8200 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'

To break TTFT into prefill computation time and transfer wait time, check the transfer latency percentile logs that vLLM periodically emits, or monitor the transfer throughput metrics exported to Prometheus via Grafana. You can also use torch.profiler to capture transfer start and completion events directly and slice the timeline from there.

Before You Commit

Check your interconnect bandwidth first. Deploying PD disaggregation on 100GbE without FP8 quantization is likely to degrade TTFT. Enabling FP8 alongside it can bring the numbers into a practical range even on Ethernet.

If your service primarily handles contexts ≤1K tokens, the prefill computation savings are small to begin with and disaggregation will yield little benefit. The same applies to low-QPS, small-batch workloads — if prefill and decode rarely conflict in a monolithic setup, there's no reason to disaggregate.

Cases where PD disaggregation on 100GbE clusters made TTFT worse are the result of skipping this analysis. Plug in two numbers — bandwidth and average context length — and the break-even calculation takes five minutes.