How Sampling Parameters Hurt LLM Serving Throughput

LLM serving environments often have an organizational division of labor around sampling parameters. A model quality engineer decides "Temperature=1.2 is what we need for creative responses," and the serving engineer takes that value and runs with it. When a p99 latency incident explodes and you trace the root cause, you sometimes land on sampling parameters. The reason is that Temperature is widely perceived as a text-quality knob, which obscures its role in serving cost.

How Temperature Shifts the Output Length Distribution

Temperature divides logits by T immediately before the softmax. When T > 1, the gap between logits narrows and the probability distribution flattens — probability spreads more evenly across the entire vocabulary. The catch is that the EOS token is not exempt from this flattening. Even when the model has internally decided "this is a good stopping point," at T=1.5 the EOS token's relative probability is diluted and is more likely to lose out to other vocabulary tokens, pushing termination further out. As a result, the same prompt set will produce longer average output lengths under T=1.5 than under T=0.7.

What's worse than the mean is the variance. At T=0.7, output lengths might cluster between 50 and 150 tokens; at T=1.5, 30-token and 800-token outputs coexist in the same batch. At T=0.0 (greedy), the highest-probability token is selected at every step, so output length variance is effectively zero. Variance grows gradually up to T=1.0, then increases nonlinearly beyond that point.

How Continuous Batching Converts Variance into Cost

Orca's (OSDI 2022) iteration-level scheduling reconstructs the batch at every decode step. The moment a request produces an EOS, its slot is released and a waiting request takes its place. This is the core mechanism that improves GPU utilization over static batching — the paper reported up to 36.9× throughput improvement over FasterTransformer.

In this architecture, high output-length variance destroys batch efficiency. If some requests in a batch finish in 50 tokens while others continue generating for 800 tokens, the short-lived slots get reused, but as long as a long-running request stays in the batch, every decode step runs on its behalf. With a batch size of 16, two or three requests generating 700+ tokens raise the TPOT ceiling for the remaining 13–14 requests.

Setting max_new_tokens=512 doesn't make this problem go away. Raising Temperature increases the fraction of requests that actually approach 512 tokens, because requests that could have terminated early encounter EOS later. The net effect is more requests surviving in the batch longer, fewer slots available for new requests, and lower throughput. Raising Temperature effectively acts as a reduction in usable batch size.

Top-p, Top-k, Min-p: Where the Decode Kernel Cost Diverges

What operation runs after the softmax determines the kernel cost of each sampling strategy.

Top-k selects the top-k tokens by probability from the vocabulary. torch.topk() uses a partial sort, so small values of k don't require sorting the entire vocabulary. Even with a 128k-token vocabulary, k=50 completes without a full sort. The apply_top_k_only path in vLLM's topk_topp_sampler.py uses this approach.

Top-p is different. To include tokens until their cumulative probability reaches p, you first have to sort probabilities in descending order and compute a cumulative sum — there is no way to know the cutoff point without it. In the vLLM source, this is implemented as logits.sort(dim=-1, descending=False) → torch.cumsum(probs_sort, dim=-1). The cost of this full sort + cumsum scales linearly with vocabulary size.

vLLM PR #11394 directly measured this overhead on the ShareGPT dataset:

The PyTorch native implementation caused ~20% performance loss in high-throughput conditions. FlashInfer uses rejection sampling to avoid the full sort entirely, effectively eliminating this overhead. Current vLLM v1 uses FlashInfer as the default path on CUDA, but falls back to PyTorch when per-request generators or post-filtering logprobs are required.

Min-p is a method proposed in 2024 (arXiv:2407.01082, ICLR 2025 Oral). Rather than a fixed threshold, it filters dynamically relative to the probability of the highest-probability token. It mitigates the problem of Top-p including too many vocabulary entries at high Temperature, and because of how it's structured, it can be computed without a full sort — keeping kernel cost close to that of Top-k.

SqueezeBits' measurements showed that enabling sampling caused vLLM to drop throughput by 15.4% and increase TPOT by 20.6% at a request rate of 8, while TensorRT-LLM saw a 7.1% throughput drop and 9.2% TPOT increase under the same conditions. The vLLM overhead being roughly 2× larger was because vLLM at that time relied on a Python-based implementation while TensorRT-LLM used custom CUDA kernels. After FlashInfer integration, this gap has narrowed considerably. The same measurements ranked individual sampling method overhead as Top-K > Top-P > repetition penalty — note that this reflects pre-FlashInfer, PyTorch-native vLLM.

presence_penalty and frequency_penalty Increase Per-Step Cost

Temperature and Top-p affect the number of decode steps, but the computation per step remains constant. presence_penalty and frequency_penalty have a different structure: at every decode step, they scan the entire set of tokens generated so far and modify the logits. At 50 output tokens, that's 50 tokens to scan; at 500 tokens, it's 500. The per-step cost grows linearly with output length.

Raising Temperature makes outputs longer, and stacking frequency_penalty on top means the later decode steps of those long outputs keep getting slower. This is why tail latency compounds when both parameters are active simultaneously — the increase in decode step count from Temperature multiplies by the increase in per-step cost from the penalty. It's worth auditing any long-form generation endpoint that has presence_penalty / frequency_penalty enabled by default.

Sampling Parameters Should Be Co-Designed with Serving Engineers

These cost pathways are invisible when teams are siloed. There's no channel through which "I set Temperature=1.5 for more creativity" gets translated into the throughput loss it causes at the serving layer. The first step in debugging a p99 latency spike is to pull the output token count distribution from your API request logs — if the variance is abnormally high, Temperature is the first thing to suspect.

Practical operating guidelines:

• Enforce a Temperature ceiling at the serving layer. In production, clamp to a maximum of 1.0; anything higher is permitted only on a dedicated experiment endpoint.
• Design max_new_tokens and Temperature together. Allowing max_new_tokens=2048 with Temperature=1.5 maximizes the worst-case batch occupancy time for that combination. Either lower the Temperature ceiling or reduce max_new_tokens — one or the other.
• Allow presence_penalty / frequency_penalty by default only on short-form generation endpoints (200 tokens or fewer). On long-form endpoints, the linear per-step cost growth in the latter half of decoding will threaten your SLOs.
• When both Top-p and Top-k are specified, filtering is the intersection of both conditions. vLLM handles this in a single kernel pass, but there's no reason to pile on unnecessary combinations.