How Chunked Prefill Trades TTFT: Latency and Throughput by Chunk Size

When p50 TTFT sits comfortably around 200ms but p99 suddenly spikes to 3–5 seconds, the culprit is almost always a long prefill request mixed into the batch. There's nothing wrong with the short requests themselves — the problem is structural: one 4096-token prefill monopolizes the GPU while seven 512-token requests queue behind it. Measuring prefill complexity on a per-request basis and then trying to predict production TTFT will always give you the wrong answer — synthetic benchmarks assume uniform request lengths, but real traffic has far greater length variance.

How Prefill Holds the GPU

In LLM serving, prefill processes the entire input sequence in a single forward pass. Even after the introduction of continuous batching, this fundamental structure hasn't changed. Even when decode requests are present in the batch, no decode step executes until the prefill forward pass completes.

If a 4096-token prefill takes 300ms, every decode request in the same batch absorbs 300ms of additional TPOT (inter-token latency). A newly arrived 512-token request can't even begin its own prefill — it just waits out those 300ms. This is what "prefill taxes decode" actually means in practice.

Iteration-level scheduling in continuous batching allows new requests to be inserted each time a decode request generates a token, but there's no insertion point until the prefill finishes. A single prefill forward pass is the scheduler's minimum unit of occupancy.

Three Bottleneck Paths

If you attribute TTFT latency purely to "long prefill," you end up looking only at attention compute cost. In practice, three paths operate simultaneously.

Attention computation itself is O(n²d) FLOPs, and using FlashAttention doesn't reduce this FLOP count. What FlashAttention-2 cuts is the number of HBM reads and writes. Standard attention writes the Q×K dot-product result (n×n elements) to HBM, then reads it back for the softmax. FlashAttention avoids materializing that n×n matrix in HBM by processing it tile-by-tile in SRAM. For N=4096 and d=128, HBM traffic drops from roughly 134MB to 17MB. But the FLOPs are identical. This is why prefill remains compute-bound even with FlashAttention.

KV cache block preemption is the second path. vLLM's PagedAttention manages the KV cache in fixed-size blocks. A 4096-token request immediately reserves 256 blocks at block_size=16. On a 24GB GPU running LLaMA-2-7B (weights ~13.5GB), roughly 10GB is available for the KV cache. At 32 layers, 32 KV heads, head_dim 128, in fp16, each block is 16 × 32 × 32 × 128 × 2 × 2 = 8MB. Total block count: 10GB / 8MB ≈ 1,250 blocks. A single 4096-token request instantly claims about 20% of the pool. Numerically, the remaining 224 blocks could accommodate seven 512-token requests (each needing 32 blocks), but the scheduler stops admitting new requests once available blocks fall below a threshold.

Memory fragmentation is the third path: repeated block allocation and deallocation leaves small gaps scattered throughout the pool. Long sequences that need contiguous allocations can't find available blocks, forcing them to wait or causing existing requests to be preempted. The impact of this path depends on block_size and the sequence length distribution.

Which path dominates depends on the environment. When a few long requests occasionally mix into otherwise short batches, attention compute cost is the primary driver. When request concurrency is high and KV cache utilization is elevated, block preemption and fragmentation move to the foreground.

How Much Does TTFT Diverge in Mixed Batches?

With vLLM at max_num_seqs=8, comparing batch compositions of (1×4096 + 7×512) versus (8×512) makes the TTFT distribution difference for 512-token requests immediately clear.

That's roughly a 5–6× difference at p99. DistServe (Zhong et al., OSDI '24) measures mixed-batch throughput on a single A100 dropping to approximately 1.6 req/s, versus 5.6 req/s for prefill-only and 10 req/s for decode-only workloads. This quantifies how much each workload degrades the other when they share a single batch.

Chunked Prefill's Trade-offs

The principle behind chunked prefill, proposed in Sarathi-Serve (Agrawal et al., OSDI '24), is straightforward: split a long prefill into fixed-size chunks and interleave decode steps between them. Splitting a 4096-token prefill into chunks of 1024 allows decode requests to interleave four times. From a 512-token request's perspective, instead of one 4096-token prefill holding the GPU in a single shot, it's now four 1024-token holds.

In vLLM, two parameters control this behavior:

vllm serve meta-llama/Llama-2-7b-chat-hf \
  --enable-chunked-prefill \
  --max-num-batched-tokens 2048

--max-num-batched-tokens sets the maximum number of tokens the scheduler processes per step, which effectively becomes the chunk size. According to the vLLM documentation, chunked prefill is enabled by default where possible in vLLM V1, with a default of 2048. For throughput-oriented workloads, 8192 or higher is recommended.

Changing chunk size moves three metrics in opposite directions:

Each time chunk size halves, short-request p99 TTFT drops by roughly half. Long-request TTFT grows proportionally with the number of chunks — because a decode step runs before each chunk is rescheduled into the batch. Throughput loss comes from the overhead of re-entering the scheduler and re-allocating KV blocks on every chunk boundary.

Sarathi-Serve measured 2.6× higher serving capacity versus vLLM after applying chunked prefill on a single A100 server with Mistral-7B. That figure rises to 3.7× on Yi-34B with 2×A100, and to 5.6× on Falcon-180B with pipeline parallelism. The larger the model and the more complex the parallelism configuration, the larger the absolute contribution of prefill-decode interference.

If your SLO is something like "p99 TTFT ≤ 500ms" centered on short-request latency, reducing chunk_size to 1024 or below is the right move. Conversely, for workloads like long-document summarization where TTFT SLOs are loose, raising it above 4096 preserves throughput. Applying the same chunk_size to a single endpoint handling both long and short requests leaves neither workload optimally served.

What P/D Disaggregation Solves — and What It Doesn't

Prefill/decode (P/D) disaggregation, which places prefill and decode on separate GPU pools, physically eliminates the structure that allows prefill to monopolize decode GPUs. Per DistServe measurements, this delivers up to 7.4× throughput improvement and 12.6× SLO tightness improvement on mixed workloads.

The catch is that transferring the KV cache produced by prefill to the decode GPU introduces a new bottleneck. Longer sequences generate more KV data to transfer, and if network bandwidth is insufficient, KV transfer latency delays the start of decoding. Where chunked prefill splits time within a single GPU, P/D disaggregation introduces inter-GPU data movement cost as the new trade-off. It's a cleaner solution to the batch interference problem in isolation, but without upfront analysis of infrastructure cost and KV transfer latency, there will be operating regimes where latency actually worsens.

Diagnosing in Production

vLLM's Prometheus metrics can pinpoint the root cause directly.

# p99 TTFT trend
histogram_quantile(0.99,
  rate(vllm:time_to_first_token_seconds_bucket[5m])
)

# Prefill time p99 — if this rises at the same time as TTFT spikes, prefill interference is the cause
histogram_quantile(0.99,
  rate(vllm:request_prefill_time_seconds_bucket[5m])
)

# Waiting request count — a sudden increase signals KV cache exhaustion or scheduler backpressure
vllm:num_requests_waiting

# KV cache utilization — above 0.8, block preemption paths begin activating
vllm:kv_cache_usage_perc

If vllm:request_prefill_time_seconds p99 rises at the same time as vllm:time_to_first_token_seconds p99, prefill is the cause. If vllm:num_requests_waiting rises alongside it, KV cache block exhaustion is also contributing.

The adjustment sequence: start with --enable-chunked-prefill and --max-num-batched-tokens tuning. If request length variance is large enough to limit the effectiveness of chunked prefill, consider length-based routing to direct short and long requests to separate instances. P/D disaggregation is the step after that.