LLM Serving Schedulers: Who Decides When Requests Hit the GPU

Dissecting the LLM Serving Scheduler: Who Decides the Order Requests Hit the GPU, and How

After introducing continuous batching, someone asked me, "Why is TTFT still spiking?" Low GPU memory? Model too large? — the cause was much closer than that. It was the scheduler policy. On the same GPU with the same model, a single flag can shift p99 TTFT by a factor of several times.

What Continuous Batching Fixed — and What It Didn't

LLM serving in the static batching era meant waiting for a batch to fill up before running inference all at once. Every sequence was padded to match the longest one, and the next batch couldn't start until every request in the current batch finished generating. GPUs were chronically underutilized under this scheme.

Orca (Yu et al., OSDI 2022) proposed iteration-level scheduling and changed the paradigm. After each forward pass, the scheduler steps in to remove completed sequences and push in new requests. The scheduling unit becomes the iteration, not the batch size. In benchmarks published by Anyscale, pure continuous batching showed an 8× throughput improvement over static batching; combined with vLLM's PagedAttention, that number reached 23×.

The problem lives just past the boundary of "what it fixed." To start the next iteration, continuous batching must wait for the current one to finish. Prefill and decode have structurally different per-iteration latencies.

How Request Length Variance Stresses the Scheduler

Prefill is compute-bound; decode is memory-bound. Even with the same GPU and the same model, per-iteration time diverges significantly between the two phases. On an A100 80GB running Llama-3 70B, a rough reference: a 4,096-token prefill takes about 120 ms, while a single decode step takes about 18 ms.

That ratio is the root cause of head-of-line blocking. During those 120 ms, decode could have advanced 6–7 steps. If there are 10 requests mid-decode in a continuous batching queue and a new 8,192-token input arrives, next-token generation for all 10 requests stalls until that prefill finishes. Under FCFS scheduling, the moment a long request takes the front of the queue, shorter requests waiting behind it receive no first token for hundreds of milliseconds.

Continuous batching works well when traffic is uniform. When request-length variance grows, p99 TTFT blows up.

Chunked Prefill: Split the Prefill and Interleave with Decode

Sarathi-Serve (Agrawal et al., OSDI 2024) attacks this problem head-on. Instead of processing a long prefill in a single iteration, it slices the prefill into chunks and interleaves them between decode iterations. An 8,192-token prefill split into chunks of 512 produces 16 chunks, distributed across decode steps. Shorter requests waiting behind get a decode opportunity after every 512 tokens of prefill are processed.

vLLM enables this with the --enable-chunked-prefill flag, and it is on by default starting from V1. SGLang lets you set the chunk size directly with --chunked-prefill-size.

# vLLM (explicit activation for V0 and earlier)
vllm serve meta-llama/Llama-3-70b-Instruct \
  --enable-chunked-prefill \
  --max-num-batched-tokens 2048

# SGLang
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3-70b-Instruct \
  --chunked-prefill-size 4096

Choosing the right chunk size is tricky. Sarathi-Serve's empirical results on Yi-34B show that below C=128, the attention kernel falls into the memory-bound regime and throughput drops sharply. Above C=2048, blocking time climbs again and the benefit of chunked prefill fades. The sweet spot was around C≈512 for Mistral-7B and C≈1024 for LLaMA-2 70B TP4. The paper calls this the "flat region" — a safe band roughly 1.5–2× wide around the optimum — so the practical goal is landing inside that band rather than pinpointing the exact peak.

The vLLM documentation states explicitly for max_num_batched_tokens: values at or below 2048 improve ITL; values at or above 8192 improve throughput. The SGLang documentation recommends --chunked-prefill-size 16384 for long-context workloads, and dropping to 4096 or 2048 under heavy memory pressure.

Scheduler Policy Comparison: FCFS, Preemption, Priority

vLLM, SGLang, and TGI all support chunked prefill, but their scheduler policies differ.

When vLLM needs to evict a running request due to KV cache pressure, it has two paths: swap the KV cache to CPU, or drop the sequence and rerun it from prefill later (recompute). Swap is bottlenecked by PCIe bandwidth. On a Llama-3 70B TP8 setup, the KV cache for an 8,000-token sequence is roughly 2.5 GB per GPU. Moving that data to CPU memory alone takes about 0.1 s, and when KV cache transfers compete with weight loading over the PCIe bus, a "convoy effect" can send P99 latency from 200 ms up to 8 s — cases like this have been reported in production. That is why vLLM V1 switched the default to recompute.

TGI's --waiting-served-ratio controls fairness from a different angle. When the ratio of waiting requests to running requests exceeds this threshold, the scheduler pauses the current decode batch and starts prefilling new requests. At the default of 1.2, the scheduler tilts toward incoming requests whenever the wait queue is more than 20% larger than the running batch. Pairing it with --max-waiting-tokens adds a hard upper bound: "force-drain the wait queue after this many tokens." Together, the two parameters cap how long long requests can monopolize decode.

SGLang's RadixAttention operates at a different layer from the scheduling policy itself. Because it stores KV caches for requests sharing a common prefix in an LRU radix tree, a cache hit means the prefill computation for that portion is skipped entirely. For workloads with repeated prefixes — multi-turn conversation, RAG with a shared system prompt — TTFT is structurally lower independent of any scheduler parameter tuning.

p99 TTFT Is Far More Sensitive to the Scheduler Than Average Throughput Is

Setting max_num_batched_tokens low limits how many prefill tokens are batched per iteration, giving decode requests more frequent scheduling opportunities. ITL and p99 TTFT improve; throughput drops. Raising the value increases throughput but also increases how often a long prefill monopolizes an entire iteration, which pushes p99 up. vLLM's documentation makes this tradeoff explicit: below 2048 for ITL improvement, above 8192 for throughput improvement.

Turning chunked prefill off maximizes throughput, but production reports have shown P99 latency spiking from 200 ms to 8 s under KV cache contention — caused by repeated events where a single long prefill request freezes the entire decode queue.

The Sarathi-Serve paper quantifies the scale of this difference. On Yi-34B TP2, chunked prefill with stall-free batching achieved 3.7× higher RPS capacity than vLLM; on Falcon-180B TP4×PP2, that figure reached 5.6×. These gains came entirely from changing the scheduler policy — no additional hardware, no model compression.

Choosing a Scheduler Based on Traffic Pattern

For chatbot-style workloads dominated by short requests with strict tail-latency SLOs, chunked prefill with max_num_batched_tokens at or below 2048 is the right call. You sacrifice some throughput to keep p99 in check.

For long-context workloads (RAG, summarization), increasing the chunk size is necessary to bring TTFT itself down. SGLang's --chunked-prefill-size 16384 is designed for this. ITL for short requests will climb somewhat at this setting.

For multi-turn conversation or workloads with a repeated system prompt, SGLang with RadixAttention has a structural advantage. When a high fraction of requests share the same prefix, migrating to SGLang typically yields much larger TTFT gains than tuning chunked-prefill parameters in any other framework — because cache hits eliminate the prefill computation entirely.