Hidden Costs of Serving Qwen3: How /think Tokens Explode Output Length Variance

When serving Qwen3 with vLLM, certain metrics start misbehaving in ways that are hard to ignore. P99 TTFT runs more than 10× slower than P50, KV cache utilization suddenly blows past 90%, and batch size drops by half even though request volume hasn't increased. Dig into the cause and it almost always comes back to a single thinking-mode request. While that request decodes tens of thousands of tokens, every other request in the batch has to wait — and that's what destroys the latency numbers.

The Qwen3 technical report (arXiv:2505.09388) shows that performance increases monotonically as thinking-chain length grows. From a serving engineer's perspective, that's exactly the problem. The more the model thinks, the longer the output; the longer the output, the more pressure on the serving stack.

What Thinking Mode Does to Output Length

Qwen3's <think> block emits the entire reasoning process before producing the actual response. The official deployment guide recommends a max_tokens of 32,768 for general tasks, rising to 38,912 for competition-level math and coding. Reasoning-optimized models like DeepSeek-R1 and QwQ routinely generate thinking chains exceeding 10,000 tokens for a single request.

Running Qwen3-8B on 100 prompts (a mix of coding, math, general conversation, and summarization) shows that the output-token distribution splits completely between thinking on and off:

Look at variance, not averages. With thinking off, nearly every request finishes under 2,000 tokens. With thinking on, p99 is more than 7× p50, and the gap between extreme and ordinary requests exceeds 10×. That variance is the direct cause of scheduler pain. A request that's 2× the median is manageable. A request that's 10× the median with p99 at 60× is a different problem entirely.

arXiv:2601.10274 models this as an M/G/1 queue. Service time scales affinely with allocated token count, but accuracy follows a diminishing-returns curve. On MATH-500, accuracy climbs from roughly 0.4 to 0.6 as thinking length grows from 0 to 5,000 tokens, but improvements drop off sharply beyond 10,000 tokens. That's why giving every request the maximum thinking budget is suboptimal from both a throughput and an accuracy standpoint.

KV Cache Occupancy and Preemption

vLLM's PagedAttention manages the KV cache in fixed-size blocks (16 tokens by default). A block is allocated for every token generated during the decode phase, so a single thinking request that produces 10,000 tokens exclusively holds 625 blocks.

With a batch size of 16 and max_tokens=8,192, the scheduler theoretically needs to reserve up to 16 × 8,192 = 131,072 tokens' worth of KV cache blocks simultaneously. The moment the GPU memory left over after model weights can no longer cover that reservation, the scheduler preempts some requests.

In vLLM V1, the default preemption strategy is RECOMPUTE — it re-runs the prefill phase instead of swapping to disk. For short requests the overhead is low, but if a thinking request has to recompute a long prefill, that cost lands directly on TTFT. Once gpu_cache_usage_perc crosses 90%, preemption frequency increases sharply.

Fragmentation is also a factor. Blocks freed by short-lived requests get interleaved with blocks belonging to long thinking requests, creating holes that are hard to use as contiguous memory. When the free-block count falls below the threshold, the scheduler stops accepting new requests and the wait queue starts building up.

Concrete Scenarios Where the Scheduler Breaks Down

Continuous batching adds a new request to the batch as soon as one finishes. But if a single thinking request keeps decoding for 20,000 tokens, one batch slot stays locked for all 20,000 steps.

The failure modes appear differently in TTFT and TPOT. For incoming requests, TTFT suffers because prefill scheduling opportunities are scarce while a thinking request is mid-decode — this is even worse when chunked prefill is disabled, since a full prefill consumes an entire batch step. For requests already in the decode phase, TPOT grows because GPU throughput gets diluted when a thinking request is occupying the batch.

This is a well-documented pattern in LLM tail latency analysis: a large fraction of cases where P99 TTFT is 5–10× P50 trace back to heavy-tailed output-length requests monopolizing the batch. That's why even 25% thinking traffic can significantly inflate overall P99 TTFT — as the proportion of thinking requests in the batch grows, tail latency deteriorates nonlinearly.

The max_tokens Dilemma

In thinking mode, setting max_tokens is a lose-lose choice.

Set it small (512–1,024) and the thinking chain gets truncated before it can close the </think> tag. vLLM's thinking_token_budget cutoff mechanism injects reasoning_end_str and transitions to the response with the reasoning incomplete. On MATH-500, shrinking the thinking budget from 5,000 tokens to 512 sends accuracy back down from 0.6 to the 0.4 range.

Set it large (8,192–32,768) and per-request KV cache reservation explodes. Every request in the batch must reserve up to max_tokens blocks for the worst case, which lowers the effective batch size ceiling and reduces throughput.

The practical range is 2,048–8,192. For a service dominated by general Q&A, 2,048 is sufficient. If competition math or complex code generation is the primary workload, 8,192 or higher is necessary, with the associated batch size and throughput penalty accepted.

Mitigation Strategies: Three Levers and Their Costs

Chunked Prefill and Parameter Tuning

In vLLM V1, enable_chunked_prefill is on by default. This option tells the scheduler to process decode requests first, then use the remaining max_num_batched_tokens budget to work through prefill in chunks. It's effective at preventing a thinking request's decode from blocking other requests' TTFT.

python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen3-8B \
    --reasoning-parser qwen3 \
    --enable-chunked-prefill \
    --max-num-batched-tokens 2048 \
    --max-num-seqs 32

max_num_batched_tokens=2048 favors better TPOT (inter-token latency). Set it too low, though, and prefill gets sliced so finely that TTFT actually increases. If throughput is the priority, raise it to 8,192 or higher. Lowering max_num_seqs reduces both concurrent request count and KV cache pressure together — if preemption is occurring frequently, this is the first knob to turn.

Capping Chain Length with thinking_token_budget

vLLM's reasoning support lets you limit the thinking-chain length per request without needing full instance separation. vLLM force-injects </think> when the reasoning token count hits the budget, terminating that phase of decode.

from vllm import LLM, SamplingParams

sampling_params = SamplingParams(
    max_tokens=4096,
    thinking_token_budget=2048,  # cap the thinking chain at 2048 tokens
)

A 2,048-token budget is sufficient for general Q&A, summarization, and translation. MATH-500-level competition math needs 5,000 tokens or more. Applying different budgets by task type is significantly better than imposing a single uniform cap.

To disable thinking entirely, pass enable_thinking=False on the request:

response = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
    messages=[...],
    max_tokens=2048,
)

General conversation, translation, summarization, and information extraction see almost no quality degradation in non-thinking mode. Multi-step math, complex algorithmic coding, and reasoning tasks that require long logical chains do suffer measurable accuracy drops when thinking is disabled. The deciding criterion is whether the task requires intermediate reasoning steps.

Separate Serving

The most effective approach is routing thinking and non-thinking requests to separate instances. When the two are never mixed in the same scheduler, the tail latency problem disappears at the root. The two workloads have different optimal operating points, which makes separate configuration worthwhile:

• Non-thinking instance: max_num_seqs=64, max_tokens=2,048 → maximize throughput
• Thinking instance: max_num_seqs=16, max_tokens=8,192–32,768 → handle long chains

The cost of separation is loading model weights twice, or managing instance counts for each pool independently as the traffic mix shifts. If your SLA is strict and thinking traffic exceeds 20%, the ROI on separation is strong. Below that, capping thinking_token_budget and tuning max_num_seqs is usually enough.

Here are the key metrics to monitor in production and the thresholds that should trigger action: