Continuous batching allows requests to be added or removed at each iteration, improving GPU utilization. Thanks to iteration-level scheduling introduced by Orca (OSDI '22), the inefficiency of request-level batching — where the entire batch had to wait until a specific request completed — was largely eliminated. But when long-context requests start appearing in the traffic mix, p99 TPOT spikes sharply. The latency contamination that continuous batching was supposed to solve reappears in a different form.
How a Single Request Stalls the Entire Batch
LLM inference is split into two phases: prefill and decode. Prefill processes all input tokens at once to populate the KV cache; decode then generates one token per iteration from that point on. The continuous batching scheduler decides batch composition before each iteration, but it cannot mix a newly arrived prefill request with requests already in the decode phase within the same iteration. Prefill requires a full forward pass over all input tokens at once.
When a prefill request arrives, the scheduler dedicates that entire iteration to it. A short input means that iteration finishes quickly, but an 8K or 32K input means that single iteration takes a very long time. The prefill phase is compute-bound and its duration scales nearly linearly with the number of tokens processed. During that time, all other requests in the decode phase are blocked, waiting for the next iteration.
This is the prefill stall. Continuous batching eliminated request-level stalls, but iteration-level stalls remained untouched.
This is exactly why p99 TPOT spikes. TPOT directly reflects iteration latency. The moment a 32K-input request enters the batch, every request currently in the decode phase absorbs that stall in full. In mixed traffic where long prefill requests arrive intermittently, the rising p99 latency tail is a structural consequence of this design.
Why Schedulers Prioritize Prefill
Typical continuous batching schedulers give prefill requests high priority. Requests in the decode phase already occupy GPU memory with their KV caches. Deferring a prefill request means it sits in the queue without holding any memory, so processing it first to populate its KV cache appears reasonable from a memory resource allocation standpoint.
From a TPOT perspective, this policy penalizes requests in the decode phase. During a prefill iteration, decode requests lose an entire iteration. Short prefills produce small stalls that rarely matter, but a long prefill lasting hundreds of milliseconds directly raises the TPOT of every decode request waiting through that iteration.
How Chunked Prefill Works
The core idea of chunked prefill, first proposed in Sarathi (Agrawal et al., 2023), is to split the prefill into fixed-size chunks and process one chunk per iteration. The remaining batch slots are filled with decode requests that run alongside — what the paper calls "piggybacking decodes with chunked prefills."
With a chunk size of 2048, a 32K prefill is spread across 16 iterations. Each iteration processes 2048 prefill tokens bundled together with tokens from the existing decode requests. Under the original approach, a single 32K iteration would have stalled the entire decode phase; with chunked prefill, the stall per iteration is bounded by the chunk size. The tradeoff is that total prefill completion time increases. Splitting 32K into 16 chunks incurs additional kernel call and attention reconstruction overhead compared to processing it all at once.
Chunks That Are Too Small Leave the GPU Idle
Transformer attention is compute-intensive due to the dense operations between tokens within a sequence. When chunks are very small — say, 256 tokens — the compute contributed by attention shrinks, and the relative overhead of kernel launches and memory accesses grows. As arithmetic intensity drops, GPU SMs become underutilized.
The Sarathi paper is the first to explicitly measure this tradeoff. There is a point where going below the optimal chunk size improves individual decode TPOT but degrades overall system throughput. In the paper's benchmarks, at the optimal chunk size, decode throughput improved by 10× for LLaMA-13B on an A6000, and 4.25× for LLaMA-33B on an A100.
How TTFT, TPOT, and Throughput Respond to Chunk Size
| Metric | Larger chunks | Smaller chunks |
|---|---|---|
| TTFT | Worse (more iterations to complete prefill) | Better |
| p99 TPOT | Better (less stall per iteration) | Worse |
| Throughput | Improves up to a point → stall returns if too large | Worse (GPU utilization drops) |
Larger chunks process more prefill tokens per iteration, which worsens TTFT. At the same time, per-iteration stall decreases, stabilizing decode p99 TPOT. Smaller chunks do the opposite — TTFT improves, but throughput suffers because the GPU is not kept sufficiently busy. These three metrics cannot be optimized simultaneously; chunk size is the single parameter that determines the balance point.
vLLM and SGLang Configuration
In vLLM V1, chunked prefill is enabled by default. The key parameter is --max-num-batched-tokens, which caps the maximum number of tokens processed per iteration and effectively sets the chunk size.
# TPOT-first — decode-heavy mixed traffic
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--max-num-batched-tokens 2048
# Throughput-first — small model on a large GPU
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--max-num-batched-tokens 8192
In V0-style usage, --enable-chunked-prefill must be specified explicitly. The vLLM documentation gives 2048 as the default and recommends 8192 or higher when running a small model on a large GPU for maximum throughput.
In SGLang, the equivalent setting is --chunked-prefill-size.
python -m sglang.launch_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--chunked-prefill-size 2048
SGLang handles prefill and decode separately by default, so chunked prefill must be explicitly enabled. For workloads dominated by inputs of 8K tokens or more, a chunk size that is too small results in a prefill-bound state. Measurements show that raising --chunked-prefill-size to 32768 yields a 34–78% improvement in output throughput and a 39–59% reduction in TTFT on 8×H200 for an 8K-input / 1K-output workload.
Serving capacity measurements from Sarathi-Serve (arXiv:2403.02310) using chunk-based stall-free scheduling: 2.6× over vLLM for Mistral-7B on a single A100, 3.7× for Yi-34B on 2×A100, and 5.6× for Falcon-180B under pipeline parallelism.
Choosing Based on Workload
Whether chunked prefill is effective depends on the input length distribution.
It has the greatest impact in patterns where most requests have short inputs — chatbots, code completion — but long-context requests appear intermittently. This is precisely where the structural contamination of the decode batch by long prefills is most severe. The larger the p95/p50 ratio of input lengths — i.e., the wider the distribution — the more valuable chunked prefill becomes.
For traffic where all requests have uniform input lengths, the effect is minimal. Everyone produces similar stalls and absorbs similar stalls, so distributing them across chunks does not change the overall distribution.
For services where prefill is inherently long — summarization, RAG — setting chunks too small significantly inflates TTFT. If TTFT SLAs are strict, it is better to increase the chunk size or process the prefill in a single iteration. When long prefills dominate and both TTFT and TPOT must be tightly controlled, it is worth considering prefill-decode disaggregation rather than chunked prefill. That scenario exceeds what scheduling within a single GPU pool can address.
KV Cache and Prefix Caching
When chunked prefill is enabled, the KV cache for a given request is built up incrementally across multiple iterations. Each time a chunk is processed, the KV blocks for that chunk are allocated. In PagedAttention-based systems, physical blocks are allocated on demand, so this incremental growth is not itself a concern.
There is a subtlety when combining chunked prefill with prefix caching. Prefix caching reuses the KV cache from a shared prefix across multiple requests, but when a common prefix is split across multiple iterations by chunked prefill, the timing at which those shared prefix blocks are completed may not align with iteration boundaries. When enabling both features in vLLM simultaneously, it is worth monitoring cache hit rate and tuning chunk size accordingly.