Why KV Cache Transfer Becomes the Bottleneck After Prefill-Decode Disaggregation

P/D disaggregation clearly works. Separating prefill and decode nodes eliminates interference between the two phases and lets you apply independently optimized parallelism strategies to each. DistServe and Splitwise have demonstrated this.

But in practice, disaggregation often underdelivers. You split the stages to reduce TTFT, and TTFT actually goes up. Or you look at GPU utilization and the decode nodes are sitting idle. The disaggregation idea itself isn't the problem — KV cache transfer is the hidden bottleneck surfacing.

KV Cache Must Move

When a prefill node processes the full input token sequence, it produces Key and Value tensors at every layer. These KV caches are needed as-is for attention computation during the decode phase, so they must be transferred to the decode node the moment prefill completes. No amount of clever scheduling avoids this transfer. It's a structural cost of disaggregated serving.

Computing KV Cache Transfer Volume

The bytes that need to move can be expressed in a single formula:

size = 2 (K+V) × L × H_kv × D × S × B × dtype_bytes

• L: number of transformer layers
• H_kv: number of KV heads (much smaller than Q heads with GQA)
• D: head dimension (head_dim)
• S: sequence length (tokens)
• B: batch size
• dtype_bytes: 2 for FP16

Plugging in Llama-3 70B (L=80, H_kv=8, D=128, FP16):

size = 2 × 80 × 8 × 128 × S × B × 2 bytes
     = 327,680 × S × B bytes

1K-token sequence, single request (B=1):

327,680 × 1,024 = 335,544,320 bytes ≈ 336 MB

Batch size 32 (B=32):

336 MB × 32 ≈ 10.7 GB

One thing worth calling out here: Llama-3 70B uses GQA (Grouped Query Attention), which gives it only 8 KV heads. Using the full 64 Q heads would overestimate transfer volume by 8×. This is exactly why you must check num_kv_heads when computing KV cache size in any real implementation.

Transfer volume scales linearly with both sequence length and batch size. In high-throughput scenarios where both are large simultaneously, the volume explodes multiplicatively.

Network Topology vs. Reality

Mapping those transfer volumes to each interconnect:

A single Llama-3 70B decode step takes roughly 20–50 ms on TP=8 A100. Put those numbers next to the table and the problem is immediately obvious.

NVLink comes in at 12 ms even at batch 32 — under one decode step. Transfer is unlikely to be a bottleneck here. IB NDR is 6.7 ms for a single request, which is acceptable, but hits 214 ms at batch 32 — 4–10× a decode step. IB HDR is already approaching a full decode step at 13 ms for a single request, and degrades quickly as batch size grows.

100GbE clocks in at 27 ms for a single 1K-token request. The decode node is burning the equivalent of an entire decode step just waiting for KV data. At batch 32, that's 856 ms — enough time to generate dozens of tokens, all spent waiting on transfer.

The critical sequence length makes this even more concrete. The point at which KV transfer time for a single request on 100GbE exceeds one decode step (~30 ms):

336 MB × (S / 1024) / 12.5 GB/s > 30ms
→ S > 30ms × 12.5 GB/s × 1024 / 336 MB
→ S > ~1,143 tokens

Any request over 1K context served over standard Ethernet will have its disaggregation gains eaten by transfer overhead.

How This Shows Up in TTFT and TPOT

Transfer latency manifests differently in TTFT and TPOT.

TTFT takes the direct hit. The window from request arrival to first token is composed of prefill time + KV transfer time + first decode step. The decode node is completely stalled until KV transfer finishes. The numbers in the table above add directly to TTFT.

TPOT is determined by compute inside the decode node, so it doesn't take a direct hit from transfer latency. However, accumulated decode node idle time prevents the scheduler from feeding in the next request, which degrades overall throughput and drives up tail latency. If you're chasing a TPOT p99 spike and look at decode node utilization, this is often where the answer is.

DistServe explicitly calls this window "migration overhead" and recommends NVLink-based same-node transfer. Mooncake goes further, redesigning the transfer path itself with a distributed KV cache store that spans CPU DRAM, SSD, and RDMA — this is what Moonshot AI's Kimi service runs on, and they've reported processing over 100 billion tokens per day across thousands of nodes.

Without measuring this gap, you will systematically overestimate the gains from disaggregated serving. When a profiler shows "Prefill X ms, Decode Y ms" and the gap between them is missing, you can't explain why the overall TTFT is what it is.

Mitigation Strategies and Trade-offs

KV Compression (INT8 Quantization)

Quantizing the FP16 KV cache to INT8 immediately before transfer cuts transfer volume in half — every number in the table above gets divided by 2. The downside is a dequantization step back to FP16 on the decode node after arrival.

vLLM's FP8 KV cache is fundamentally different: it stores KV in FP8 in GPU memory at all times. Storage quantization is applied persistently to reduce memory pressure; transfer compression is applied only during the transfer window to reduce network bandwidth. You can use both together, but the accuracy loss from transfer compression is a separate concern from storage quantization.

Chunking and Pipelining

Splitting the transfer into per-layer chunks lets the decode node start attention computation for earlier layers the moment their KV arrives, overlapping transfer latency with compute. This is a direction Splitwise explores. Implementation complexity goes up, but it can meaningfully reduce TTFT in long-sequence settings.

Topology Selection

The most reliable fix is handling prefill and decode within nodes connected by NVLink. H100 DGX intra-node NVLink bandwidth is 900 GB/s — under 12 ms even at batch 32. Crossing racks over IB blows that number up by 10× or more.

That said, NVLink configurations require splitting GPUs within a single server, which makes it harder to independently scale the number of prefill and decode GPUs. Multi-node IB configurations allow elastic scaling but carry the bandwidth constraint. Operational flexibility and transfer cost are in direct tension.

When Disaggregation Actually Wins

Once you factor in transfer cost, the conditions under which disaggregated serving comes out ahead are narrower than you might expect.

In environments serving short context at high frequency — chatbot queries averaging 128 tokens or fewer — transfer overhead completely cancels out the disaggregation benefit. Before adopting disaggregated serving, pull the histogram of your request length distribution first.

How to Actually Measure KV Transfer Bottleneck

Without extracting the gap between prefill completion and first decode token as a separate metric, transfer bottleneck is buried inside the TTFT number. The most direct approach is to expose both timestamps as Prometheus metrics.

import time
from prometheus_client import Gauge

kv_transfer_gap = Gauge(
    "llm_kv_transfer_gap_seconds",
    "Gap between prefill completion and first decode step",
    ["request_id"]
)

# Prefill node: immediately before KV transfer begins
prefill_done_ts = time.monotonic()

# Decode node: record the gap immediately before first forward pass
kv_transfer_gap.labels(request_id=req_id).set(
    time.monotonic() - prefill_done_ts
)

To inspect at the NCCL level, use nsys:

nsys profile \
  --trace=cuda,nvtx,nccl \
  --output=kv_transfer_profile \
  python serve.py

The Send / Recv timestamps in the NCCL trace break down how many milliseconds KV transfer takes per layer. On InfiniBand, run ib_send_bw / ib_read_bw first to confirm actual bandwidth, then compare against the theoretical numbers above.

If KV transfer gap accounts for more than 20% of TTFT, that system is giving back a significant portion of its disaggregation gains — and will continue to do so until the network topology or batching strategy changes.