Why LLM Serving Has Long p99 Latency Tails

If you work on serving infrastructure, this situation is probably familiar. Grafana shows a healthy mean TTFT of 400 ms, but the moment you expand the p99 panel, you see spikes past 3 seconds. GPU memory utilization is sitting around 60% and everything else looks fine. You try lowering max_num_seqs, and throughput drops from 3,320 tok/s to 2,180 tok/s — but p99 collapses from 3,241 ms to 852 ms. That's the opposite of what you'd expect. The root cause is in how the scheduler constructs batches.

Why p99 degrades several times faster than the mean

In a continuous batching setup, the scheduler decides each iteration whether to admit new requests into the current batch. Orca (Yu et al., 2022) introduced iteration-level scheduling to fix the poor GPU utilization of static batching, but it left behind a new structural problem. When a long request is in the batch, short new requests can't enter and have to wait in the queue while that long request continues decoding.

Say max_num_seqs=32 and the batch is already full at 32 requests. A newly arriving request waits until one slot opens up. If the batch contains a request that needs to output 1,024 tokens, the queue keeps growing until that request finishes. This is Head-of-Line (HOL) Blocking — structurally identical to the phenomenon in networking where a single packet stalls an entire stream.

This is exactly why p50 and p99 can diverge by several times. Most requests enter the batch quickly, but requests that arrive right when long requests have monopolized the batch must wait for all of them to finish. Under uniform traffic, the probability of hitting that window is low, so the p50–p99 gap is modest. Under bursty traffic or with high variance in request-length distributions, long requests hold the batch for extended periods more frequently, and p99 grows non-linearly. More GPU memory doesn't fix this. Unless you change how many requests and how many total tokens the scheduler admits per batch, HOL Blocking frequency stays the same regardless of how much headroom you have.

Three scheduler parameters pull three metrics in different directions

The parameters in vLLM that directly control batch composition are max_num_seqs and max_num_batched_tokens.

max_num_seqs is the maximum number of requests processed per iteration. Increasing it grows the batch size, which raises parallelism in GPU matrix operations and improves throughput. At the same time, it intensifies contention for batch slots, driving up TTFT p99 for new requests. More requests decoding simultaneously means more KV cache memory pressure and a higher probability of preemption.

max_num_batched_tokens is the maximum total number of tokens processed per iteration. When chunked prefill is enabled, this parameter directly affects TTFT. Lowering it makes prefill chunks smaller, which reduces the inter-token latency impact that prefill has on decode batches. The trade-off is that smaller chunks reduce GPU SM utilization. vLLM's default for chunked prefill is 2,048 — tuned toward ITL optimization — so if throughput is your primary goal, you'll likely need a higher value.

Preemption is what happens when the KV cache runs out: the scheduler suspends an in-flight request and frees its cache. When it fires, that request's TPOT spikes sharply. Sarathi-Serve's measurements show that simply having a prefill interleave into a batch can push inter-token latency up to 28× higher than a decode-only batch. vLLM supports two preemption modes — recompute (suspend and recompute from scratch) and swap (offload to CPU) — and from a latency standpoint recompute is preferable, since swap's CPU transfer delay shows up directly as a TPOT spike.

How each parameter change moves each metric:

Measurement: sweeping max_num_seqs while tracking all three metrics

On an A100 80 GB running Llama-3-8B with a fixed input length of 512 tokens, I swept max_num_seqs from 8 to 64 under two scenarios: uniform output lengths (fixed 256 tokens) and mixed output lengths (uniform distribution from 64 to 1,024 tokens).

Uniform output (fixed 256 tokens)

Going from max_num_seqs=32 to 64, throughput increased only 8% (3,320 → 3,580 tok/s). Meanwhile, TTFT p99 went up 3.8× (852 ms → 3,241 ms) and TPOT p99 went up 5.5× (89 ms → 487 ms). That's a 4–6× sacrifice in tail latency for an 8% throughput gain.

The widening gap between p50 and p99 makes the problem even more concrete. At max_num_seqs=8, the TTFT spread is 56 ms (168 − 112). At max_num_seqs=64, it's 2,405 ms (3,241 − 836). As the batch grows larger, lucky and unlucky requests have increasingly extreme experiences.

The inflection point is around max_num_seqs=32. Up to that point, doubling max_num_seqs roughly doubled throughput (1,240 → 2,180 → 3,320) — the GPU still had headroom and larger batches were translating efficiently into more work done. Beyond 32, throughput gains flatten while p99 shoots up. The measurements confirm that this inflection is determined by scheduler queue depth, not GPU memory availability: from 32 to 64, GPU memory utilization changed by less than 3%, but preemptions went from 0 to 12 per minute.

Mixed output traffic (max_num_seqs=32, fixed)

At the same max_num_seqs=32, mixed traffic produces TTFT p99 that is 4.3× higher than uniform traffic. TPOT p99 is more than 10× higher.

Why high variance in request lengths causes non-linear tail latency explosion

Under uniform traffic at max_num_seqs=32, requests in the batch finish decoding at roughly the same pace and cycle out together. New requests quickly fill the open slots, and any HOL Blocking is short-lived.

Mixed traffic behaves very differently. While a 64-token request finishes and leaves the batch, a 1,024-token request is still occupying its slot. New arrivals wait for open slots, but long requests hold onto them far longer, so turnover is slow. More requests pile up in the queue, and those waiting requests in turn encounter more long requests ahead of them, extending their wait further. This feedback loop is non-linear: a small increase in the fraction of long requests causes a disproportionately large jump in queue wait time.

In production, when the standard deviation of your request-length distribution is large, the gap between the p99 you measured in a lab with synthetic uniform traffic and the p99 you actually see can be much larger than you'd predict. That's why you need to check your request-length distribution before trusting any benchmark numbers.

Sarathi-Serve attacks this problem with chunked prefill. By splitting prefill into small chunks and mixing them with decode batches, no single long prefill can monopolize the batch all at once. Here are TTFT p99 and GPU SM utilization measurements at max_num_seqs=64 across different max_num_batched_tokens (chunk size) settings:

Capping chunks at 512 tokens brings TTFT p99 down to 245 ms, but GPU SM utilization drops to 58%. Leaving it unlimited keeps the GPU at 95% but pushes p99 to 3,241 ms. vLLM's chunked prefill default of 2,048 sits somewhere in between — and that value may not be optimal for your workload either.

Define your SLO first, then set the parameters

A service with an SLO of TTFT < 200 ms and a service with an SLO of p99 TPOT < 50 ms need to move their parameters in opposite directions. If TTFT is your primary constraint, lower max_num_seqs (target 8–16) and keep max_num_batched_tokens small (512–1,024). This minimizes the time requests spend waiting to enter the batch — and the costs are lower GPU utilization and reduced throughput.

If p99 TPOT is your primary constraint, decode continuity matters most. Keep max_num_seqs in the 16–24 range and set the preemption policy to recompute. Swap is worse here because the CPU transfer delay shows up directly as a TPOT spike.

A practical tuning sequence: if TTFT p99 starts breaching your SLO, cut max_num_batched_tokens in half and observe. If TPOT p99 starts breaching your SLO, reduce max_num_seqs by 4 at a time while monitoring preemption counts. If preemptions are already at zero but TPOT p99 is still high, look at the request-length distribution inside the batch. If length variance is the cause, request-length-aware routing or priority scheduling will be more effective than parameter tuning alone.

The numbers in this table are based on Llama-3-8B on an A100 80 GB. Larger models or GPUs with different memory bandwidth will shift the critical max_num_seqs threshold. The right approach is to track which metric hits its SLO bound first, then decide which parameter to move and in which direction based on that signal. The specific numbers only come from sweeping them yourself.