Why TTFT, TPOT, and p99 Latency Move Independently in LLM Serving

TTFT is 200 ms, but p99 is 1.2 s. p50 latency looks fine, yet p99 is firing alerts. To diagnose whether this is a server configuration problem or a specific request-pattern issue, you need to understand what each of the three numbers — TTFT, TPOT, and throughput — actually measures, and why they can move independently of one another.

TTFT (Time to First Token) is the time from when a request arrives until the first token is delivered to the client. TPOT (Time Per Output Token) is the average interval between subsequent generated tokens. Throughput is the number of requests (req/s) or tokens (tok/s) the server processes per unit time. These three don't always move in the same direction because the prefill and decode stages are bottlenecked by different GPU resources.

TTFT is bound by prefill compute; TPOT is bound by memory bandwidth

LLM inference has two stages. Prefill processes the entire input prompt to build the KV cache. Decode then reads that cache and generates output tokens one at a time.

Prefill is compute-bound. Self-attention FLOPs scale as N² with input length N — a 2,048-token prompt requires 64× more compute than a 256-token prompt. The A100 80GB peaks at 312 TFLOPS in FP16, and prefill nearly saturates that compute capacity. A high TTFT almost always means "prefill took too long."

Decode is different. Every time a token is generated, the KV cache for every transformer layer must be read from HBM. Per-token FLOPs are tiny, but the amount of memory that must be read scales with the total KV cache size of all sequences in the batch. The A100's 2 TB/s HBM bandwidth is the real bottleneck here, and TPOT is determined by how much of that bandwidth remains available.

The fact that the two stages are bottlenecked by different resources creates a structural tension. One approach to reducing TTFT is chunked prefill — splitting prefill into small chunks and interleaving them with decode steps. Agrawal et al. SARATHI (2023) demonstrated up to 10× higher decode throughput on LLaMA-13B with this approach, and reduced pipeline-parallel stalls (bubbles) by 6.29× in GPT-3-scale settings, yielding a 1.91× overall throughput improvement. However, because chunked prefill spreads a single prefill across multiple steps, TTFT for that individual request actually increases.

Why does TTFT increase as batch size grows?

Iteration-level scheduling — continuous batching — introduced by Yu et al. Orca (OSDI 2022) inserts new requests into an active decode batch at every token step. Anyscale's benchmarks showed up to 23× throughput improvement over naïve static batching.

As throughput climbs, TTFT for incoming requests comes under pressure. If the server preempts ongoing decodes to run prefill immediately on a new request, TTFT stays low but existing decode requests are blocked. If instead the server keeps decoding and queues the prefill, decode continuity is preserved but the new request's TTFT grows by however long it waits. Scheduler policy tries to balance these two, but as the GPU approaches saturation, both suffer.

The relationship between batch size and TTFT is nonlinear. The table below shows representative patterns for Llama-3.1-8B on an A100 80GB using the ShareGPT length distribution.

p50 TTFT rises 43% from 1 to 8 concurrent requests; p99 TTFT jumps 166% over the same range. Throughput increases by more than 600%. Pushing parameters toward maximum throughput degrades p99 latency nonlinearly.

Why p99 diverges from p50: heavy-tail request length distribution

The average input in the ShareGPT dataset is 202 tokens; the average output is 179 tokens. In real traffic, this distribution has a heavy tail — the top 1% of requests can run into thousands of tokens. Understanding what happens when one of those long requests lands in a batch is the key to understanding p99.

During decode, the batch stays together until the longest sequence finishes. If a single 2,048-output-token request is in the batch, all the 128-token requests that finish their generation early still hold their slots until that long request completes — only then does a slot open for a new request's prefill. During that window, new requests wait in the queue. This is head-of-line blocking.

vLLM handles this via preemption: it swaps the long request's KV cache to CPU memory or recomputes it to free a slot, then processes the waiting request's prefill first. Throughput continues, but the preempted request pays a swap I/O or recomputation penalty, and that overhead feeds back into the batch's overall TPOT. TGI takes a different approach with the max_total_tokens parameter: requests whose combined input + output length exceeds the threshold are either blocked from entering the queue or returned as errors. This keeps p99 low but narrows coverage for long-context requests.

Uniform-length benchmarks (e.g., fixed 512/512 tokens) hide this phenomenon entirely. When all sequences in a batch are the same length, blocking never occurs and p99 measures nearly the same as p50. Trusting those numbers in production is how you end up with p99 TTFT spiking 3–5× under real traffic.

Three ways benchmarks diverge from reality

Uniform-length requests. Fixed-length inputs and outputs underestimate tail latency. No distribution means no tail; no tail means no head-of-line blocking. Meaningful measurements require length distributions drawn from ShareGPT or actual service logs.

Low concurrency. One to eight concurrent requests do not saturate the GPU. Throughput measured under these conditions is optimistic, and scheduler contention is invisible. You need to sweep request rate upward in steps to find where the saturation knee bends.

Cold measurements without warmup. The first few dozen requests trigger CUDA kernel compilation, KV cache initialization, and other one-time costs — TTFT can read in the seconds range during this period. A mean TTFT that includes this window will be several times higher than steady-state latency. Run enough warmup requests and exclude that window from your measurements.

A reproducible measurement design

Separate what you fix from what you vary.

Fixed: input length distribution (ShareGPT or real logs), model and quantization level, GPU count, warmup exclusion criteria, and measurement window.

Sweep: request rate (0.5 → 1 → 2 → 4 → 8 req/s). At each point, record p50, p95, and p99 separately. Measure throughput at the point just before the saturation knee — throughput numbers past the knee look impressive, but at that point p99 TTFT is already at a level that makes the service unusable.

# vLLM benchmark CLI (v0.6.0+)
vllm bench serve \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --dataset-name sharegpt \
  --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 500 \
  --request-rate 2 \
  --percentile-metrics ttft,tpot,itl \
  --metric-percentiles 50,95,99

ray-project/llmperf can produce the same saturation curve by specifying ShareGPT as the input dataset and stepping up concurrent workers incrementally.

If you don't report p50, p95, and p99 separately, the tail stays hidden. Reporting a single mean TTFT is only meaningful under the assumption that request length is uniformly distributed.

Which metric to prioritize depends on your use case. For chatbots and interactive interfaces, a TTFT under 200 ms is the practical target. For real-time streaming applications like code autocomplete, staying below 30–50 ms TPOT determines perceived quality. For batch processing pipelines, the throughput at the saturation knee matters more than TTFT itself. Without deciding in advance where to give headroom among these three numbers, you'll keep hitting the same situation: bumping max_num_batched_tokens lifts the throughput metric while simultaneously triggering TTFT alerts.