Prefill-Decode Disaggregation: Why Mixing Both Phases on One GPU Pool Hurts

LLM inference processes a single request through two fundamentally different compute phases on the GPU. Prefill processes all input tokens at once; decode produces output tokens one at a time, sequentially. On the surface they both run the same transformer operations, but they consume GPU resources in fundamentally different ways. Without understanding this distinction, decisions about chunked prefill parameters and when disaggregation is warranted end up being guesswork.

Arithmetic Intensity: Why the Two Phases Differ

Where GPU performance bottlenecks occur is determined by arithmetic intensity — the ratio of FLOPs to bytes of memory traffic. When a workload reads more data from memory relative to the compute it performs, it is memory-bandwidth-bound; when it performs more compute relative to memory reads, it is compute-bound. On the A100 80GB SXM4, BF16 throughput is 312 TFLOPS and memory bandwidth is roughly 2 TB/s (2,039 GB/s), putting the ridge point at about 156 FLOP/Byte. Arithmetic intensity above this number is compute-bound; below it is memory-bandwidth-bound.

During the prefill phase, all tokens in the input sequence are processed simultaneously through matrix multiplications. As token count T grows, the matrix dimensions grow and arithmetic intensity rises with them. Looking at just one layer's attention QKV projection for a 1,024-token prefill, it is straightforward to exceed tens of FLOP/Byte. Prefilling a sufficiently long sequence runs the A100 in the compute-bound regime.

Decode is an entirely different picture. Generating a single output token requires reading every layer's weight matrices from memory and reading the KV cache for every token generated so far. At batch size 1, this reduces to a matrix-vector multiplication, keeping arithmetic intensity around 1–2 FLOP/Byte — roughly 100× below the 156 FLOP/Byte ridge point. Most of the GPU's compute units sit idle, waiting on memory transfers. Increasing batch size helps, but KV cache size puts pressure on VRAM, so the decode batch cannot grow without bound.

This asymmetry is the starting point for every design decision that follows.

What Happens When Both Phases Share the Same GPU Pool

Before continuous batching, prefill for one request had to complete before moving to the next iteration, so a long prefill would simply stall every other request's decode. Continuous batching addressed this by recomposing the batch each iteration, but it did not eliminate the underlying problem.

The issue is that iterations are not uniform in length. A single decode iteration completes in a few milliseconds; prefilling a long sequence can consume tens to hundreds of milliseconds per iteration. When a prefill request enters the batch, all requests that were in the middle of decode cannot receive their next token until the prefill iteration finishes. The TPOT (Time Per Output Token) for those requests spikes by exactly the length of that prefill iteration.

In a real-time chat service, this manifests as streaming output flowing smoothly, then freezing, then suddenly resuming. Monitoring TPOT p99 shows an acceptable average while the tail distribution is severely inflated.

The effect runs in the other direction too. When the decode batch is full and a new request arrives, its prefill is delayed — TTFT (Time To First Token) increases. The end result is that a heavy mix of long prefill requests degrades both TTFT and TPOT simultaneously. The fact that continuous batching increases throughput and the fact that prefill-decode interference still occurs within it are not contradictory — throughput and latency tail distribution are separate metrics.

Chunked Prefill: Spreading the Work Across Time

The core idea is straightforward. Rather than processing a prefill all at once, it is split into fixed-size chunks and interleaved between decode iterations. A 4,096-token prefill is handled as eight passes of 512 tokens each, with a decode iteration inserted between each chunk. The long "decode-blocking" event caused by a large prefill becomes many shorter interruptions spread across time.

Sarathi-Serve (Agrawal et al., OSDI 2024) formalized this approach. Measured as the number of requests that can be served while meeting the same TBT (Time Between Tokens) p99 SLO, it outperforms vLLM by:

Chunk size is a dial between TTFT and TPOT. Smaller values mean each prefill chunk disrupts decode less, improving TPOT stability, but the same prefill now spans more iterations, increasing TTFT. Larger values have the opposite effect. In Sarathi's experiments, using C=512 token chunks added less than 3% computational overhead, and the optimal chunk size varied with model size — around 512 tokens for Mistral-7B and around 1,024 tokens for LLaMA2-70B.

In vLLM, chunked prefill is enabled with --enable-chunked-prefill and the chunk size is controlled via --max-num-batched-tokens.

# chunk size 2048 — vLLM current default (ITL optimized)
vllm serve meta-llama/Llama-3-8B \
  --enable-chunked-prefill \
  --max-num-batched-tokens 2048

# TPOT stability first — small chunk
vllm serve meta-llama/Llama-3-8B \
  --enable-chunked-prefill \
  --max-num-batched-tokens 512

# TTFT first, or throughput first — large chunk
# (for small model + large GPU combinations, 8192+ is recommended)
vllm serve meta-llama/Llama-3-8B \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192

Per the vLLM documentation, the current default is 2,048 tokens (it was 512 through v0.4.2). This default is not always optimal. If requests are mostly short conversations, decode batch size has more impact than chunk size. If requests are primarily multi-thousand-token document summarization, even the default chunk size can produce a noticeable TTFT increase.

SGLang takes a different approach. Its overlap scheduler pipelines prefill chunks and decode batches on the same GPU, running them concurrently. Where vLLM executes prefill chunk → decode sequentially, SGLang runs some decode post-processing in parallel with the next prefill chunk. This theoretically improves throughput, but the different implementation complexity and memory access patterns mean the relative advantage shifts depending on the model and configuration.

Prefill-Decode Disaggregation: Separating into Different GPU Pools

If chunked prefill distributes the interference across time, disaggregation eliminates the interference entirely. The GPU pool for prefill and the GPU pool for decode are fully separated, and the KV cache produced after prefill completes is transferred over the network to the decode nodes.

DistServe (Zhong et al., OSDI 2024) studied this architecture systematically. With the same number of GPUs, it handles up to 7.4× more requests than vLLM, or meets SLOs 12.6× more stringent at the same request rate. Breaking this down by workload: 2.0× on OPT-13B ShareGPT, 4.6× on OPT-175B, and 12.6× on long document summarization (LongBench) — the gains grow with sequence length. KV cache transfer overhead was under 0.1% of total latency in the OPT-175B experiments, with over 95% of requests experiencing KV transfer latency under 30 ms.

Mooncake (Qin et al., 2024 — FAST 2025 Best Paper) is a production deployment of this architecture at Kimi. Switching from 20 monolithic instances to a 10 prefill + 10 decode disaggregated configuration delivered 75% more requests on the same GPU count. Where vLLM met the TBT SLO only 57% of the time, Mooncake achieved near 100%. The setup used RDMA at 800 Gbps interconnect, with achieved transfer bandwidth of 87 GB/s single-path and 190 GB/s bonded.

The prerequisites deserve attention here. KV cache transfer overhead staying at the 0.1% level requires RDMA high-speed interconnect. Transferring KV cache over standard Ethernet can make the transfer latency exceed the prefill time itself. Also, short sequences produce small KV caches with low absolute transfer cost, but workloads dominated by long-context sequences — thousands of tokens — can find that transfer cost becomes a new bottleneck. Part of why Mooncake was particularly effective is that Kimi's workload is heavily long-context.

Choosing the Right Approach

For a single node or small cluster, chunked prefill is almost always the practical first choice. It takes one flag and requires no additional infrastructure. Disaggregation is worth considering when RDMA interconnect is already in place and the cluster is large enough to justify managing separate prefill-only and decode-only node pools. The results in the Mooncake paper come from a production environment at Kimi's scale — a small team that adopts the same architecture wholesale will likely find that operational overhead exceeds the benefit.

Pitfalls That Are Easy to Miss in Production

When prefill-decode interference is severe, two metrics rise together: prefill queue depth (number of prefill requests waiting) and decode stall rate (fraction of decode iterations taking longer than expected). Both rising simultaneously is a signal that prefill is crowding out decode.

Using the Prometheus metrics exposed by vLLM:

# Number of requests currently in prefill
vllm:num_prefill_tokens_requests

# Scheduler queue depth (waiting before prefill is admitted)
vllm:num_waiting_requests

# Output tokens per second (proxy for decode throughput)
rate(vllm:generation_tokens_total[1m])

If prefill queue depth and num_waiting_requests are both climbing while the generation_tokens_total rate is falling, prefill load is eating into decode throughput.

Using chunked prefill together with prefix caching requires care. A prefix cache hit reuses the KV cache for that prefix, skipping its prefill computation — but if chunk boundaries do not align with prefix cache block boundaries, hit rates can drop. vLLM aligns chunk boundaries to prefix cache block sizes, but a poor combination of settings can degrade cache efficiency.

KV cache preemption also becomes a problem as prefill lengths grow. If VRAM is insufficient when starting a new request's prefill, the KV cache of an already-decoding request must be swapped to CPU or recomputed from scratch. That recomputation cost equals the original prefill cost. When preemptions occur frequently, reducing --max-num-seqs (the maximum number of sequences processed concurrently) is usually more effective than reducing chunk size. Chunk size controls the prefill load per iteration; it does not reduce total VRAM occupancy.