Cost Breakdown: Token Budget vs Early Stopping vs Truncation in LLM Serving

What Happens When You Force Shorter Outputs in LLM Serving: Comparing the Costs of Token Budget, Early Stopping, and Truncation

Operators reach for max_new_tokens for a simple reason: long outputs tie up the GPU. TPOT (Time Per Output Token) scales linearly with the number of requests being decoded concurrently, and the longer a sequence holds its KV cache, the longer the next request waits in the queue. Capping at 128 to cycle through batches faster seems perfectly reasonable.

In practice, this intuition sometimes backfires.

KV Block Reservation and the Scheduler's Arithmetic

The moment vLLM admits a request to the running queue, it needs to secure KV blocks. The number it uses as a reference is max_new_tokens. The scheduler checks whether enough free blocks exist for input token count + max_new_tokens, and if so, accepts the request. With the default block size of 16 tokens, max_new_tokens=128 reserves 8 blocks and max_new_tokens=512 reserves 32.

Setting max_new_tokens lower reduces the block reservation requirement, so the scheduler admits more requests concurrently — the batch grows. But if any request's actual output exceeds that value, vLLM preempts the sequence: it stops the in-flight sequence, evicts its KV cache, and sends it back to the waiting queue. On retry, every KV cache entry computed up to that point has to be recomputed from scratch.

When multiple requests hit their block limit simultaneously, preemptions cascade. The scheduler's optimistic admission of many requests is what triggers the cascade in the first place. Sarathi-Serve (Agrawal et al., OSDI 2024) proposes chunked prefill as a way to mitigate this, but it doesn't address the root cause — error in predicting output length.

The Cost of Preemption

Preemption is expensive because of recomputation. Replaying an 8,000-token prefill from scratch takes 0.2–0.4 seconds. The swap approach moves the KV cache to CPU memory and restores it later, but even with PCIe 4.0 x16 bandwidth (~32 GB/s), the transfer time is non-trivial for long contexts. The default mode in vLLM v1 is recompute.

In production measurements, once GPU cache utilization crosses 90% and preemptions begin, p99 TPOT can spike from 200 ms to 8 seconds. P99 reaches 3.8× P50, with roughly 70% of that gap attributable to preemption.

The worst case is the death spiral. Preempted requests return to the queue and compete with new arrivals, only to be preempted again. The result is 100% GPU utilization with throughput completely stalled.

Cost Structure of Three Strategies

There are three main approaches to controlling output length.

EOS-based early stopping is cleanest in theory. Blocks are released the moment EOS is emitted, so there's no waste. The catch is that the scheduler still needs an upper bound when admitting the request — if it falls back to max_model_len because output length is unknown, concurrent admission actually drops.

Post-hoc truncation wastes the most resources. KV blocks stay occupied through the full 2,000-token generation, then only the first 512 tokens are returned. All GPU compute and memory are consumed; only the result is clipped.

Why Budget Forcing Is Different for Reasoning Models

Cutting thinking tokens in Qwen3 or DeepSeek-R1-series models degrades quality in a different way than ordinary truncation.

These models complete chain-of-thought reasoning inside a <think>...</think> block and then generate answer tokens. Budget forcing inserts the </think> delimiter before the thinking block has finished, forcing a transition to the answer phase on top of incomplete reasoning. For a math problem, this is equivalent to guessing the final answer after abandoning the intermediate work.

On DeepSeek-R1-32B, accuracy increases by roughly +3.2% for every additional 500 thinking tokens up front. This effect saturates around 12,000 tokens and then turns negative. Cutting blindly at 512, without knowing this curve, discards exactly the range where reasoning is most productive.

The The Coupling Tax paper attributes this phenomenon to thinking and answer tokens competing for a shared total output budget. When both sides draw from the same pool, neither gets enough.

Qwen3 natively supports a thinking_budget parameter. Used together with the /think switch, the model has been trained to complete its reasoning within the budget, so quality loss is far smaller than forcing a hard cut externally. DeepSeek-R1 has no native budget control. The only option is external truncation, and quality degrades linearly the earlier below 12,000 tokens the cut happens.

Choosing a Strategy by Workload

If the output length distribution has a coefficient of variation (CV = std / mean) below 0.3, setting max_new_tokens to 1.5–2× the mean output will produce almost no preemptions. This applies to workloads with consistent output lengths, such as RAG-based summarization or classification label extraction — cases where the scheduler can reserve blocks close to what is actually needed.

For workloads where CV exceeds 1.0 — reasoning, code generation, open-ended chat — no static value can satisfy both throughput and quality simultaneously.

One approach is to set max_new_tokens generously (1.5–2× the p90 output length) while reducing max_num_seqs to cap concurrent admission directly. This converts preemption pressure into queuing latency, which at least makes p99 predictable.

Another approach is to place a length prediction model upstream and set max_new_tokens dynamically per request. As demonstrated in Proxy Model-based Sequence Length Prediction, using a small proxy model to estimate output length upfront lets the scheduler reserve blocks far more accurately. The preemption reduction from this approach is most pronounced on high-CV workloads.