Why LLM Serving Needs Separate Prefill and Decode Stages

Prefill and Decode happen inside the same transformer model, but from the GPU's perspective they are entirely different operations. Prefill processes all prompt tokens at once as a matrix × matrix multiplication; Decode generates one token at a time as a matrix × vector multiplication. This structural difference splits the two phases into fundamentally different hardware bottlenecks, and that split drives every serving architecture decision.

Prefill is compute-bound; Decode is memory-bandwidth-bound

Arithmetic intensity is the number of operations performed per byte read from memory (FLOP/byte). Every GPU has a ridge point: above it, the workload is compute-bound; below it, memory-bandwidth-bound.

The A100's ridge point is roughly 153 FLOP/byte — peak FP16 throughput of 312 TFLOPS divided by memory bandwidth of 2,039 GB/s. For a Llama-family model (hidden dim 4096), the arithmetic intensities of the two phases look like this:

Prefill sits 2–4× above the ridge point, so the GPU's compute capacity is the bottleneck. Decode, even at a batch size of 32, reaches only about 20% of 153. During Decode, the GPU spends most of its time reloading weights from memory; the actual matrix multiplications finish within that load time. This is why SM utilization can exceed 90% during a Prefill step, then drop below 20% the moment Decode begins.

No single GPU configuration optimally serves both phases simultaneously. Settings tuned for Prefill simply hand Decode idle compute it cannot use.

Interference when both phases share the same GPU

Since continuous batching was introduced, it is natural for prefilling requests and decoding requests to land in the same batch. When most prompts are short, this interference is negligible. The problem surfaces the moment a single long-prompt request enters the batch.

A request with a 2,048-token prompt monopolizes most of the GPU's compute for that step. All other requests already in the Decode phase stall until that step completes. This is where TPOT (Time Per Output Token) p99 spikes from tens of milliseconds to hundreds — one request blocking the entire batch, classic head-of-line blocking.

TTFT (Time To First Token) and TPOT are governed by different SLOs. TTFT measures how long a new request waits for its first token; TPOT measures the interval between tokens for a request already generating. Because both metrics compete for the same GPU resources, speeding up Prefill to reduce TTFT degrades TPOT, and prioritizing Decode to protect TPOT raises TTFT. Continuous batching does not resolve this fundamental trade-off.

DistServe (Zhong et al., OSDI 2024) formalized this as "prefill-decoding interference." The paper's experiments showed that existing systems trying to meet tight SLOs on both TTFT and TPOT had no choice but to over-provision GPUs or sacrifice one metric entirely.

Chunked Prefill: reducing interference within a single instance

The idea behind Chunked Prefill is straightforward. Instead of processing a long prompt in one step, split it into small chunks and interleave them between Decode steps. A 2,048-token prompt gets divided into four 512-token chunks spread across four steps. Each step's Prefill chunk is smaller, so Decode requests are blocked less.

Sarathi-Serve (Agrawal et al., OSDI 2024) systematized this as "stall-free batching," demonstrating 2.6× throughput improvement over vLLM on Mistral-7B with a single A100, and 3.7× on Yi-34B with two A100s.

In vLLM, enable it with the --enable-chunked-prefill flag and control chunk size with --max-num-batched-tokens. It is on by default starting with the V1 engine.

vllm serve meta-llama/Llama-3-8B \
  --enable-chunked-prefill \
  --max-num-batched-tokens 2048

Trade-offs by --max-num-batched-tokens value:

Smaller chunks stabilize TPOT but reduce Prefill throughput. Spreading Prefill across multiple steps means the KV cache must stay resident in memory throughout, and per-step overhead accumulates. The optimal value depends on model size and GPU memory bandwidth, so the same setting cannot be used for both 7B and 70B models.

Chunked Prefill still has Prefill and Decode sharing the same GPU — it reduces interference but does not eliminate it. When prompts are extremely long (8K tokens or more), or when both TTFT and TPOT SLOs are strict to within tens of milliseconds, this compromise is insufficient.

Disaggregated Prefill-Decode: separate instances entirely

Disaggregated Serving splits the system into a dedicated Prefill GPU pool and a dedicated Decode GPU pool, transferring completed KV caches over the network to the Decode instances. Each pool can be independently optimized for its own bottleneck: the Prefill pool maximizes compute utilization; the Decode pool maximizes memory bandwidth.

DistServe demonstrated that this architecture handles 7.4× more requests than existing systems, or achieves 12.6× tighter SLOs at the same throughput. The Kimi team's Mooncake (arXiv 2407.00079) handled 75% more requests with the same GPU count in production, with simulated scenarios showing up to 525% throughput improvement.

Current major implementations:

The prerequisite for this architecture is that KV cache transfer time must be shorter than Prefill processing time. For Llama-3 8B, a 2,048-token KV cache is roughly 270 MB; for 70B, roughly 670 MB. Transferring that data over InfiniBand HDR (theoretical bandwidth 200 Gb/s) takes tens of milliseconds. For short-prompt workloads where Prefill itself finishes in a few milliseconds, transfer cost dominates. For small models (7B or fewer parameters) or traffic where prompts are mostly 512 tokens or shorter, Disaggregated Serving costs more than it gains.

Choosing the right strategy

Disaggregated Serving carries significant operational complexity: dual GPU pools, KV cache transfer infrastructure, and inter-pool routing logic. It is only worth choosing when you must hit tight SLOs on both TTFT and TPOT simultaneously and prompts are long enough to justify the KV cache transfer overhead. Outside those conditions, Chunked Prefill delivers far better return on complexity.

Start by identifying where the bottleneck is

Regardless of which strategy you choose, measure first. Collecting TTFT and TPOT at p50/p99 usually reveals where the problem lies.

If only TTFT p99 is high and TPOT looks normal, Prefill requests are backing up in the queue. If TPOT p99 spikes intermittently while TTFT is fine, that is the classic interference pattern: Prefill intruding during Decode. In that case, enabling --enable-chunked-prefill with an appropriate --max-num-batched-tokens will substantially improve TPOT p99.

Looking at the GPU timeline in torch.profiler or NVIDIA Nsight Systems makes the SM utilization gap between Prefill and Decode steps immediately visible. If you see SM utilization above 90% during Prefill steps repeatedly followed by below 20% during Decode steps, the Decode pool is sitting idle, memory-bound. Replacing the Decode instances' GPUs with a model that has higher memory bandwidth — for example, moving from A100 (2,039 GB/s) to H100 (3,350 GB/s) — improves TPOT far more directly than upgrading the Prefill-side GPUs.