Continuous vs Static Batching: Throughput Gains and When They Reverse

Static batching's problems start with padding. Sequences in a batch don't have the same length in practice, so pad tokens are appended to shorter sequences to match the longest one. The GPU spends compute on pad tokens but produces no useful output.

Consider a batch with a 10× length variance — 32 requests mixed together, ranging from 128 to 1,280 tokens. The effective compute ratio looks like this:

유효 토큰    = sum(실제 길이) ≈ 평균 700 × 32 = 22,400
총 슬롯      = 최대 길이 × 배치 크기 = 1,280 × 32 = 40,960
실효 연산 비율 = 22,400 / 40,960 ≈ 54.7 %

Nearly half of all compute is wasted. Worse, even after shorter requests finish, the GPU can't start the next batch until the longest sequence completes.

What Iteration-Level Scheduling Is

The idea proposed in Orca (Yu et al., OSDI 2022) is to shift the scheduling unit from request to iteration. In traditional scheduling, once a batch is formed, its membership is fixed until every sequence in it finishes. In iteration-level scheduling, the scheduler intervenes after each forward pass — removing completed sequences and inserting new waiting requests.

# Static batching — request-level
batch = queue.pop(N)
outputs = model.generate(batch)   # 전체 배치가 EOS까지 대기
queue.push(next_N_requests)

# Continuous batching — iteration-level
while queue or running:
    for seq in list(running):
        if seq.is_done():
            running.remove(seq)
            if queue:
                running.add(queue.pop())
    outputs = model.forward_one_step(running)  # 토큰 하나씩

During the decode phase, one iteration is a pass that generates one token per sequence in the batch. The moment a sequence emits an EOS, that slot is reused in the very same iteration. Because the GPU never has to wait for the entire batch to finish, utilization stays high even when request lengths vary widely. This structural difference is why the Orca paper reported 36.9× throughput over NVIDIA FasterTransformer on GPT-3 175B.

How Much Throughput Actually Improves

Anyscale's 2023 benchmark puts concrete numbers on this difference. Llama-13B on A100:

That 23× figure comes from a scenario with maximum request-length variance. As the distribution becomes more uniform, the gap narrows. When all requests are the same length, static batching produces no padding at all, and vLLM's per-iteration batch reconstruction starts to show measurable scheduling overhead.

Real-world LLM workloads approximate an exponential distribution — most requests are short, with a few very long ones. Continuous batching's advantage is greatest under this distribution. You can compare the two scenarios directly using vLLM's benchmark_serving.py:

# 균일 분포 — static과 차이 작음
python benchmarks/benchmark_serving.py \
    --backend vllm \
    --model meta-llama/Llama-3-8B-Instruct \
    --dataset-name random \
    --random-input-len 128 \
    --random-output-len 128 \
    --num-prompts 500

# 실세계 분포 근사 (ShareGPT) — continuous batching 이점 극대화
python benchmarks/benchmark_serving.py \
    --backend vllm \
    --model meta-llama/Llama-3-8B-Instruct \
    --dataset-name sharegpt \
    --num-prompts 500

The throughput difference between these two runs gives you a baseline for predicting continuous batching gains on your own workload. Run this measurement before trusting any externally published benchmark numbers.

Head-of-Line Blocking

Throughput goes up, but a latency problem comes with it.

When a long sequence occupying a slot is still decoding, a newly arrived short request can't begin prefill — it has to wait until that long sequence finishes its remaining decode iterations and frees the slot. If one of the 32 batch slots is a 4,096-token request on its 3,800th decode iteration, a new 50-token request has to wait through 296 more iterations before it can even start prefill. The wait time is several times longer than the actual processing time for that 50-token request.

If your p50 TTFT is low but p99 is in the seconds, this pattern is the likely culprit. GPU SM utilization alone won't expose the problem — 90% utilization doesn't tell you whether that's one long request monopolizing the batch or many requests running in parallel.

TGI addresses this with the waiting_served_ratio parameter: a heuristic that prioritizes short requests when the number of waiting requests exceeds a certain multiple of the number of running requests. vLLM takes the approach of capping max_num_seqs to limit how many sequences can be in the batch simultaneously. Both frameworks expose the same underlying problem through different parameters.

The Upper Bound on Batch Size: KV Cache Contention

Increasing batch size raises throughput, but the KV cache sets a hard memory ceiling. Each sequence requires the following KV cache per token:

KV per token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element

Llama-3 8B (FP16, num_layers=32, num_kv_heads=8, head_dim=128):

= 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes ≈ 128 KB / token

On an A100 80GB with vLLM's default setting (gpu_memory_utilization=0.9):

가용 메모리  = 80 × 0.9 = 72 GB
가중치 점유 ≈ 16 GB  (FP16 8B 모델)
KV 캐시 풀  ≈ 56 GB

4K 컨텍스트: 56 GB / (128 KB × 4,096) ≈ 109 시퀀스
8K 컨텍스트: 56 GB / (128 KB × 8,192) ≈  54 시퀀스

Llama-3 70B weighs in at 140 GB of weights alone in FP16 (70B × 2 bytes). Two A100 80GB cards (160 GB) leave almost nothing for the KV cache. A four-card setup (320 GB) is where the KV cache pool becomes viable. Llama-3 70B has num_layers=80 and num_kv_heads=8, which pushes KV per token to 320 KB:

KV 캐시 풀  ≈ 320 × 0.9 − 140 = 148 GB
4K 컨텍스트: 148 GB / (320 KB × 4,096) ≈ 116 시퀀스

When the batch is full and KV cache runs out, vLLM swaps some sequences' KV blocks out to CPU memory. PCIe bandwidth becomes the bottleneck, causing latency spikes on the affected sequences. If max_num_seqs isn't set to match the computed capacity, these spikes will consistently pollute p99.

Chunked Prefill: What Changes When You Split Prefill

Head-of-Line Blocking describes decode sequences blocking new requests, but there's an analogous problem in the other direction. Prefilling a long prompt is compute-intensive, and all decode requests stall until that prefill iteration finishes. A 4,096-token prefill running in a single pass delays the TBT (Time Between Tokens) of every decode sequence in the batch by that full duration.

Sarathi-Serve (Agrawal et al., OSDI 2024) proposed chunked prefill: split the prefill into small chunks and interleave them between decode iterations. Processing 512 tokens per iteration from a 4,096-token prefill means decode requests share each iteration with a prefill chunk. The maximum time decode is stalled is reduced to the cost of a single chunk.

In vLLM, enable it with:

vllm serve meta-llama/Llama-3-8B-Instruct \
    --enable-chunked-prefill \
    --max-num-batched-tokens 2048

--max-num-batched-tokens sets the cap on tokens processed per iteration — the chunk size. A smaller value stabilizes decode TBT but requires more iterations to complete a long prefill, which can increase TTFT for long prompts. A larger value lets long prompts finish faster but brings back TBT jitter. Values of 512, 1024, and 4096 each strike a different throughput/latency tradeoff. For chatbot-style workloads with short prompts and long generation, 1024 is a reasonable starting point. For prefill-heavy workloads like long document summarization, 4096 is the better direction. The optimal value depends on your workload distribution — fixing a single value without measuring is not recommended.

When Continuous Batching Is a Net Loss, and What to Measure

The gains disappear when request-length variance is small. If all incoming requests are the same length, static batching produces no padding, and the per-iteration scheduling cost of continuous batching becomes pure overhead. Prefill-only workloads with no decode phase — such as embedding generation or classification — are in the same boat. For offline batch inference where it's fine to dedicate a GPU to a single request, the complex scheduler is unnecessary to begin with.

The three metrics to track separately are TTFT, TPOT, and p99 broken down by length bucket. TTFT is a direct signal of Head-of-Line Blocking. TPOT (Time Per Output Token) measures per-token decode latency, which grows with batch size as memory bandwidth contention increases. Both metrics should be tracked as independent p99 values across input-length buckets (0–512 / 512–2048 / 2048+ tokens). Without knowing whether your overall p99 is coming from a specific length range, you can't tell where a max_num_seqs adjustment actually had an effect. Evaluating chunked prefill tuning is impossible without this breakdown.