Why LLM Serving Benchmarks Lie: 3 Traps of Synthetic Traffic

If a benchmark shows 2,000 tok/s but that number has never reproduced in production, the problem isn't how you ran the benchmark tool — it's that the benchmark inputs are structurally different from your production workload. A parameter issue is fixable with a one-line change. An input design issue means the benchmark is measuring an entirely different phenomenon.

This post won't walk through how to use vLLM's benchmark_serving, SGLang's bench, or llmperf. Instead, it digs into three specific ways synthetic traffic diverges from real workloads, and shows how much each one distorts the numbers.

The Typical Betrayal Pattern

Running vllm bench serve --request-rate inf in staging drives GPU utilization above 95% and produces impressive tok/s numbers. Deploy the same model to production and p99 TPOT is more than 3× what the benchmark showed, with TTFT spiking to several seconds during peak periods.

This pattern repeats because benchmarks get three things wrong at once.

Trap 1: Request Length Distribution — Heavy Tails Clog the Decode Queue

Most LLM serving benchmarks use a fixed length or narrow distribution, like --random-input-len 512. Even using real conversation data like ShareGPT doesn't automatically make the benchmark realistic — the evaluation section of the vLLM SOSP 2023 paper shows a mean input length of 161 tokens for ShareGPT, which is already dominated by short, homogeneous requests.

Production input length distributions look different. Short requests (under 50 tokens) account for 60–70% of traffic, with sparse 2,000–8,000 token requests in the tail. Extracting from actual logs typically shows a p50 around 120 tokens, a p99 around 2,048 tokens, and a p99.9 well above that. This distribution is closer to log-normal than Gaussian — it has a heavy tail.

The problem is that even under continuous batching, these tail requests drag the entire batch during the decode phase. Prefill is parallelized, so even long prompts are processed quickly. But decode is autoregressive — one token at a time — so a single request generating 2,000 output tokens affects the TPOT of every other request in the batch for as long as it's running. vLLM and SGLang partially mitigate this with preemption, but they don't eliminate it.

A uniform 512-token benchmark never surfaces this behavior. Every request decodes at roughly the same speed, so the batch stays balanced. This is the primary reason p99 TPOT in production runs 3–5× higher than p50.

Extracting the distribution from production logs is straightforward:

import json
import pandas as pd
import matplotlib.pyplot as plt

# vLLM access log는 기본적으로 JSON lines 형식
# {"prompt_tokens": 142, "completion_tokens": 89, ...}
records = []
with open("vllm_access.log") as f:
    for line in f:
        try:
            records.append(json.loads(line))
        except json.JSONDecodeError:
            pass

df = pd.DataFrame(records)
print(df["prompt_tokens"].describe(percentiles=[.5, .9, .95, .99]))

df["prompt_tokens"].hist(bins=100, log=True)
plt.xlabel("Input tokens")
plt.ylabel("Count (log scale)")
plt.title("Production request length distribution")
plt.savefig("input_dist.png", dpi=150)

Looking at this histogram is a prerequisite for asking "what distribution should my benchmark use?" in any meaningful way.

Trap 2: Arrival Process — Closed-Loop Load Generation Artificially Flattens Traffic

Firing hundreds of requests simultaneously with --request-rate inf is a closed-loop setup — all requests are injected at the start, and no new requests arrive faster than the server can process them. Production is open-loop. Requests arrive randomly from users and keep coming regardless of server state.

Anyscale's llmperf design writeup makes this distinction explicit: a closed-loop generator only sends the next request after the previous one completes. When the server is overloaded, this automatically reduces the incoming request rate, which makes tail latency look better than it is. That's not a measurement — it's the tool self-limiting the load.

Queuing theory makes the gap concrete. In a simple M/M/1 queue, mean waiting time is:

E[W] = ρ / (μ(1 - ρ))

where ρ = λ/μ (server utilization), λ = arrival rate, and μ = service rate. Comparing wait times at different utilization levels:

Closed-loop benchmarks artificially keep utilization low. While N in-flight requests are being processed, no new ones arrive, so the queue stays short. In open-loop conditions where the arrival rate reaches 90% of service capacity, TTFT is theoretically 9× higher than the closed-loop number.

The vLLM CLI simulates a Poisson arrival process by specifying --request-rate as requests per second and adding --burstiness 1.0:

vllm bench serve \
  --backend vllm \
  --model meta-llama/Llama-3.1-8B \
  --dataset-name sharegpt \
  --dataset-path ShareGPT_V3.json \
  --request-rate 10 \
  --burstiness 1.0 \
  --max-concurrency 200

Setting --burstiness 0.1 simulates bursty traffic; --burstiness 5.0 or higher approximates uniform arrivals.

llmperf is open-loop by design. It maintains a specified concurrency level by immediately replacing each completed request with a new one, sustaining high utilization and measuring tail latency honestly.

SGLang's bench_serving.py supports the same open-loop Poisson arrival mode via --request-rate. All three tools support open-loop operation, but since the distinction isn't emphasized in defaults or documentation, it's easy to miss.

Trap 3: Prefix Reuse Rate — Cache Hit Rate Inflates Throughput Significantly

RAG pipelines and agent services with fixed system prompts share identical prefixes of hundreds to thousands of tokens across requests. vLLM's prefix caching and SGLang's RadixAttention reuse the KV cache for these prefixes, dramatically reducing prefill cost.

The problem is that benchmarks push this hit rate to one extreme or the other. --dataset-name random gives every request a completely different prompt — 0% hit rate. Conversely, a configuration like --prefix-repetition-prefix-len 1024 --prefix-repetition-num-prefixes 1 yields effectively 100% hit rate. Production sits somewhere in between.

The throughput impact by hit rate, roughly:

The actual numbers depend heavily on the ratio of prefix length to decode length. For a request with a 512-token prefix and 64 output tokens, a 100% hit rate eliminates prefill almost entirely, which can multiply throughput severalfold. If decode is 512 tokens and the prefix is short, the prefill savings matter much less.

Reporting a 0% hit rate benchmark when you don't know your actual reuse rate understates throughput. Reporting 100% produces numbers that can't be reproduced in production. Either way, a throughput number without a stated hit rate has no baseline for comparison.

Designing a Benchmark That Actually Means Something

Minimum requirements for a benchmark that addresses all three traps:

1. Input length distribution: derive from production logs

• Use the Python snippet above to extract the prompt_tokens distribution from production logs
• Capture p50, p90, and p99
• Use actual ShareGPT samples or real log samples in the benchmark — avoid putting a single value into --random-input-len

2. Arrival pattern: open-loop Poisson

• Specify --request-rate explicitly (using inf approximates closed-loop)
• Use --burstiness 1.0 for a Poisson process
• Sweep target utilization across multiple levels — 70%, 85%, 95%. A single RPS number tells you very little

3. Prefix reuse rate: separate experiments per scenario

• Run fresh (0% hit rate), estimated production hit rate, and 100% as separate experiments
• Always report the hit rate

4. Exclude the warmup period

• Drop the first 30–60 seconds (or the first few hundred requests) — the KV cache is cold and will skew results

5. Report p50 / p95 / p99 together

• Reporting only mean TTFT hides tail latency. If p99 is more than 3× p50, that discrepancy is itself a diagnostic signal

Key parameter mapping across tools:

What Numbers to Look at Together

Without reporting TTFT, TPOT, and throughput simultaneously, the results can't be interpreted. Settings that maximize throughput typically increase TTFT, and reducing TTFT requires lowering concurrency, which reduces throughput. Because the two objectives trade off against each other, optimizing for only one and reporting it tells you nothing about how the system behaves along the other axis in production.

GPU utilization and KV cache usage should also be recorded. High p99 TTFT at under 60% GPU utilization points to a request length distribution problem or prefill queue skew. The same high p99 at 95%+ utilization is a capacity problem. The fix is different in each case.

Throughput saturation experiments and latency-constrained experiments (e.g., sustaining p99 TPOT ≤ 100ms) must be run separately. The common "X tok/s at p99 TPOT 80ms" single-point report is just one point on a trade-off curve. Without knowing the utilization at which that number was achieved, and how quickly p99 degrades as load increases, the number has no practical meaning.