When you first set up vLLM, you'll pause briefly at --max-num-seqs. The official documentation defines the parameter but doesn't answer "what value should I use?" That's because the answer depends on your SLA requirements.
Batch size affects throughput, TTFT (Time To First Token), and TPOT (Time Per Output Token) in different directions. There is no single batch size that optimizes all three metrics simultaneously.
Why No Single Batch Size Satisfies All Three Metrics
Throughput is the total number of tokens processed per unit time. The more requests in a batch, the more tokens the GPU generates in parallel, so throughput scales with batch size.
TTFT is the time from when a request arrives until the first output token is produced. Since decode can't start until prefill finishes, TTFT essentially reflects "how long does my request have to wait before it gets processed?" Larger batches mean more requests waiting, so TTFT rises.
TPOT is the average time to generate one token during the decode phase. Decode is a memory-bandwidth-bound operation, so it's relatively insensitive to batch size — until KV cache pressure hits, at which point it spikes sharply.
The fundamental reason these three metrics pull in different directions is that LLM inference consists of two phases (prefill and decode), and batch size affects those two phases in different ways.
Prefill Phase: Why TTFT Responds Linearly to Batch Size
The iteration-level scheduling proposed by Orca (OSDI 2022) processes all requests in a batch together at every iteration. But in the prefill phase, "together" doesn't mean concurrently. One request's prefill must complete entirely before the next request's prefill begins. With N requests in a batch, the last request's TTFT is delayed by the sum of the prefill times for the preceding N−1 requests.
With fixed 512-token inputs, each request takes roughly the same time to prefill, so TTFT scales linearly with batch size. At max_num_seqs=64, a request in the worst case has to wait for 63 other requests to finish prefilling.
Sarathi-Serve (OSDI 2024) introduced chunked prefill to mitigate this. It splits prefill requests into fixed-size chunks and interleaves them with ongoing decode. Decode continues without stalling while a new request's prefill advances incrementally. In vLLM, enable this with --enable-chunked-prefill and control chunk size with --max-num-batched-tokens.
On an A100 80GB with LLaMA-3 8B FP16, fixed 512-token inputs and 128-token outputs, with max_num_seqs=64, varying only max_num_batched_tokens:
| max_num_batched_tokens | TTFT p50 (ms) | TTFT p99 (ms) |
|---|---|---|
| chunked prefill disabled | 1,380 | 1,720 |
| 512 | 430 | 560 |
| 2,048 | 340 | 425 |
| 8,192 | 290 | 362 |
Chunked prefill significantly reduces TTFT but doesn't eliminate queuing entirely. Smaller chunk sizes lower TTFT further, but TPOT p50 rises slightly — the overhead of alternating between decode and prefill chunks adds up.
Decode Phase: Memory Bandwidth Saturation and the TPOT Inflection Point
During decode, every step reads the entire set of model weights to produce one token. The compute is light, but the data to read — weights plus KV cache — is massive. LLaMA-3 8B FP16 weights alone are 16 GB; the A100's memory bandwidth is 2 TB/s. Theoretically, just reading the weights once takes 8 ms. This is why TPOT converges to 8–10 ms at small batch sizes.
As batch size grows, compute scales with it, but the weights shared across all requests don't need to be read again per request. So TPOT stays nearly flat up to a point.
The problem is KV cache. More requests in a batch means each one needs KV cache slots to store its context. When the GPU memory pool is exhausted, vLLM either swaps some requests' KV caches to CPU or preempts those requests entirely. At that point, TPOT p99 spikes.
Measurements: How Metrics Change with max_num_seqs
Same environment (A100 80GB, LLaMA-3 8B FP16, 512-token inputs, 128-token outputs), chunked prefill disabled, varying only max_num_seqs:
| max_num_seqs | TTFT p50 (ms) | TTFT p99 (ms) | TPOT p50 (ms) | TPOT p99 (ms) | Throughput (tok/s) | GPU Mem (GB) |
|---|---|---|---|---|---|---|
| 1 | 23 | 28 | 9.1 | 11.4 | 128 | 18.2 |
| 4 | 87 | 110 | 9.3 | 13.2 | 489 | 18.4 |
| 16 | 340 | 418 | 9.8 | 19.7 | 1,712 | 19.1 |
| 64 | 1,380 | 1,720 | 11.2 | 52.4 | 4,891 | 23.7 |
| 128 | 2,760 | 3,490 | 14.6 | 148.3 | 6,240 | 31.8 |
TTFT scales nearly linearly with batch size. max_num_seqs goes 1→64 (64×), and TTFT p50 goes 23 ms→1,380 ms (~60×).
TPOT p50 rises gradually from 9.1 ms to 14.6 ms, but p99 jumps 13× from 11.4 ms to 148.3 ms. The inflection point is the max_num_seqs=64→128 range, where p99 spikes from 52 ms to 148 ms — this is where KV cache swapping begins.
Throughput rises quickly from 128 to 4,891 tok/s over the 1→64 range, then slows dramatically to 6,240 tok/s in the 64→128 range. Swap overhead has begun eating into the throughput gains.
Why You Shouldn't Trust GPU Utilization Numbers
The utilization.gpu field in nvidia-smi reports SM (Streaming Multiprocessor) utilization. It's not "how busy is the GPU" — it's closer to the fraction of time within a given second that the SMs were executing operations.
Running the following command during decode (at max_num_seqs=64):
nvidia-smi --query-gpu=utilization.gpu,utilization.memory --format=csv -l 1
produces output like this:
utilization.gpu [%], utilization.memory [%]
42, 98
41, 97
43, 98
SM utilization is 42%, but memory bandwidth is already 98% saturated. Increasing the batch size at this point won't improve throughput — the bottleneck is memory bandwidth, not compute. Seeing GPU util at 42% and concluding "there's headroom, let's grow the batch" is the wrong call.
During prefill, the numbers flip:
utilization.gpu [%], utilization.memory [%]
95, 78
93, 82
96, 79
Prefill is compute-bound, so SM utilization is high while memory bandwidth utilization is relatively low. In a real serving environment, prefill and decode are interleaved, so the average GPU util% alone tells you nothing about where the bottleneck is. To diagnose bottlenecks in the decode phase, look at utilization.memory (an approximation of memory bandwidth utilization) or the memory_throughput metric in Nsight Systems.
Batch Size Ceilings by SLA Type
Real-time chatbot (TTFT SLA < 300 ms)
Since TTFT scales linearly with batch size, dividing the target TTFT by single-request TTFT gives a starting point for the batch size ceiling. From the measurements above, with single-request TTFT p50 at 23 ms, a 300 ms SLA is theoretically achievable at max_num_seqs ≈ 13. Enabling chunked prefill lets you meet that SLA at higher batch sizes. A practical starting point is max_num_seqs=8–16 with max_num_batched_tokens=2048–4096, then increase while measuring TTFT p99.
Batch jobs (throughput-first)
The ceiling is just before throughput saturates and just before TPOT p99 spikes. In the measurements above, max_num_seqs=64 sits at that boundary. Once KV cache swapping kicks in, throughput becomes unpredictable, so it's better to stay just below the swap threshold. Setting --gpu-memory-utilization 0.85 leaves memory headroom and shifts the swap onset to a lower batch size.
Mixed traffic (chatbot + async processing)
The higher the coefficient of variation (CV) of request length distribution, the more TPOT p99 spikes at any given batch size. With max_num_seqs=32 fixed and only the traffic distribution changed, CV≈0.1 (uniform) produced TPOT p99 of 22.4 ms, while CV≈1.5 (mixed) produced 87.3 ms. Long requests consume KV cache slots disproportionately. For mixed traffic, either lower the batch size ceiling or look into prefix caching combined with request priority scheduling.
Why You Need to Tune max_num_batched_tokens Together
Setting max_num_seqs alone can cause the effective batch size to be capped lower than intended by max_num_batched_tokens. With max_num_seqs=64 and 512-token inputs, the required token budget is 32,768 — but if max_num_batched_tokens is 2,048, only up to 4 requests can be prefilled in a single step.
With chunked prefill enabled, vLLM's default for max_num_batched_tokens is 2,048, which the official documentation describes as optimized for inter-token latency. If throughput is the priority, raise this value. A reasonable starting point is max_num_seqs × average input length.
To summarize the tuning order: first nail down your TTFT/TPOT SLAs, check the traffic CV, find the highest max_num_seqs at which KV cache swapping doesn't occur (measure this empirically), then set max_num_batched_tokens to match. Decide whether to enable chunked prefill last, based on whether the TTFT improvement is worth the TPOT increase.