Pipeline Parallel Scheduling: 1F1B, Interleaved, and Zero-Bubble Tradeoffs

GPU 8장에 Pipeline Parallel을 걸었는데 nvidia-smi를 보면 SM utilization이 30%대를 헤매는 경우가 있다. 스테이지를 늘렸더니 오히려 GPU가 더 놀고 있다. 이 현상의 이름은 파이프라인 버블이고, 스테이지 수가 늘수록 필연적으로 심해지는 구조적 문제다.

Why Bubbles Form

Pipeline Parallel distributes model layers sequentially across p GPUs. Stage 2 cannot start until a batch has passed through stage 1's layers. The timeline with a single batch:

GPU 1 (stage 1): [F1][  idle  ][B1][  idle  ]
GPU 2 (stage 2):     [F2][  idle  ][B2][ idle ]
GPU 3 (stage 3):         [F3][ idle ][B3]
GPU 4 (stage 4):             [F4][B4]

GPU 1 finishes F1 and immediately stalls, waiting for the forward passes on GPUs 2–4 to complete. It stalls again waiting for the backward pass to propagate back to stage 1. These idle intervals are the bubbles. Assuming uniform stage execution time t, the bubble ratio at m=1 is:

bubble_ratio = (p-1) / (m + p - 1)

At m=1 this reduces to (p-1)/p. With p=4 that's 75%; with p=8 it's 87.5%. Eight GPUs attached, and on average seven of them are idle.

This formula is the standard notation defined in Narayanan et al. (2021). The numerator (p-1) counts the idle steps during pipeline fill (front) and drain (back); the denominator (m+p-1) counts the total microbatch processing steps.

GPipe: Fill the Pipeline with Microbatches

GPipe's solution is to split a batch into m microbatches. As soon as stage 1 passes microbatch 1 downstream, it immediately starts processing microbatch 2. In the steady state, when the pipeline is fully occupied, every stage is simultaneously processing a different microbatch. Bubbles are confined to the fill phase (p-1 steps) and the drain phase (p-1 steps).

With p=4 and m=16: bubble ratio = 3/19 ≈ 0.16. With p=4 and m=4: 3/7 ≈ 0.43. As m grows, the denominator grows and the ratio approaches zero.

The catch is memory. GPipe must keep the intermediate activations for all m microbatches in memory simultaneously to support the backward pass. Activation memory scales as O(m×p). For Llama-3 70B (hidden size 8192, 80 layers) with p=4, each stage holds 20 layers. At m=16, each stage must buffer activations for 16 microbatches concurrently—tight even on an A100 80GB.

1F1B: Fix Activation Memory at O(p)

1F1B (one-forward-one-backward) was first proposed in PipeDream (Microsoft Research, SOSP 2019) and later adapted by Megatron-LM for synchronous training.

GPipe runs all m forward passes before starting any backward pass. 1F1B starts the backward pass as soon as the pipeline fill completes. Once stage p finishes the forward pass for microbatch 1, stage 1 immediately begins that microbatch's backward pass. From that point on, each stage alternates: one forward, one backward.

The bubble ratio is the same as GPipe:

bubble_ratio_1f1b = (p-1) / (m + p - 1)

However, at any given moment the number of microbatches simultaneously live in the pipeline is capped at p. Activation memory is fixed at O(p)—a factor of m/p savings over GPipe. At m=16 and p=4, that's a 4× reduction.

In Megatron-LM this is implemented as forward_backward_pipelining_without_interleaving in schedules.py. The --num-micro-batches argument controls m.

Interleaved 1F1B: Split Stages to Cut Bubbles Further

The only way to reduce 1F1B's bubble ratio without increasing v is to increase m. But a larger m means a larger total batch size and higher per-microbatch latency. Interleaved 1F1B attacks a different axis.

Instead of each GPU owning one contiguous chunk of layers, each GPU owns v non-contiguous chunks. With p=4 and v=2, GPU 1 handles layers 1–10 and 41–50; GPU 2 handles layers 11–20 and 51–60. The number of logical stages becomes p×v = 8, but the physical GPU count remains 4.

Bubble ratio:

bubble_ratio_interleaved = (p-1) / (m*v + p - 1)

With p=4, m=8, v=2: 3/(16+3) = 3/19 ≈ 0.158. Basic 1F1B at m=8 gives 3/11 ≈ 0.273, so this is more than a 40% reduction. Same m, lower bubble ratio.

The cost is communication. Each microbatch traversing the pipeline incurs v times as many inter-stage activation transfers. The P2P communication that occurs (p-1) times per microbatch in basic 1F1B becomes v(p-1) times.

For Llama-3 70B with hidden size 8192 and sequence length 2048, the activation tensor transferred between stages is 2048 × 8192 × 2 bytes (bfloat16) ≈ 32 MB. With p=4, m=16, v=4, the inter-stage transfer count per step is 4 × 3 × 16 = 192, for a total data volume of 192 × 32 MB ≈ 6 GB.

NVLink supports ~600 GB/s bidirectional per GPU, so this communication takes only a few milliseconds. On a PCIe-based cluster the bandwidth narrows to roughly 64 GB/s bidirectional and the same transfers can cost tens of milliseconds. The crossover point where increasing v becomes counterproductive is approximately:

t_comm × v(p-1) × m > Δbubble_time

where t_comm = activation_size / bandwidth and Δbubble_time = (bubble_ratio_1f1b - bubble_ratio_interleaved_v) × T_total. On PCIe clusters, even v=2 can violate this condition.

In Megatron-LM, --num-layers-per-virtual-pipeline-stage controls v. In DeepSpeed, use pipeline_num_chunks.

Zero-Bubble Is a Training-Only Strategy

The Zero-Bubble (ZB) schedule from Qi et al. (2024) splits the backward pass into two phases: B (input gradient) and W (weight gradient). B must propagate activation gradients to the previous stage and therefore must respect ordering constraints, but W only updates the weights of the current layer and can be deferred freely. ZB-H1 fills bubble intervals with W operations, theoretically driving the bubble ratio to zero.

Deferring W in the training pipeline shifts the timing of weight updates. The paper reports no measurable convergence difference experimentally. For inference there is no backward pass, so this strategy does not apply.

Inference Pipeline Bubbles Look Different

In training, pipeline bubbles arise entirely from forward-backward asymmetry and pipeline fill/drain. Inference is a one-directional pipeline with no backward pass, so the bubble structure is simpler.

The complication is request non-uniformity. Prefill (prompt processing) and decode (token generation) have very different execution times. Prefill scales linearly with prompt length; decode processes one token per step. When both types coexist in the pipeline, stage execution times diverge. Fast stages stall waiting for slow ones—a new form of bubble.

Bursty arrivals make this worse. While stage 1 is backlogged with new requests, stages 3 and 4 have nothing to process. Conversely, a single long-context request that takes a long time at stage 1 creates a reverse bottleneck where all downstream stages wait. The SGLang pipeline parallelism roadmap explicitly identifies this inter-stage imbalance as a key improvement target.

Combining prefill-decode (PD) disaggregation with pipeline parallelism makes the problem more complex, since the two types have different stage occupancy patterns. PD disaggregation combined with pipeline parallelism is currently an active development area in SGLang.

Theoretical Bubble Ratio vs. Observed GPU Utilization

The table below shows the theoretical values from the GPipe/1F1B formula (p-1)/(m+p-1) alongside estimated GPU utilization observed on an A100 80GB NVLink cluster running Llama-3 70B.

Observed GPU utilization runs slightly below (1 - bubble_ratio) due to inter-stage communication, scheduler overhead, and activation checkpoint recomputation. On NVLink the gap is a few percentage points. On PCIe-based clusters, communication overhead eats an additional 5–15 pp.

Interleaved schedule comparison (p=4, m=16):

With v=2 on NVLink, the communication cost is comfortably smaller than the bubble reduction benefit, yielding reliable gains. Starting at v=4, the benefit shrinks even on NVLink; on PCIe clusters, throughput often regresses.

Practical Decision Criteria

Network topology comes first. On NVLink clusters, v=2 interleaved is nearly free; treat v=4 conservatively until you have empirical numbers. On PCIe clusters, increase m before touching interleaved—it's much safer.

If memory is tight, 1F1B has a clear edge over GPipe. Same bubble ratio, but activation memory is fixed at O(p).

When combining with Tensor Parallel, minimize the number of pipeline stages. TP already introduces intra-node all-reduce traffic; adding pipeline stages piles cross-node P2P communication on top. The standard recommendation is to set the TP degree equal to the number of GPUs per node (typically 8) and only increase pipeline stages when the model exceeds single-node memory capacity.

For inference-only pipelines where inter-stage imbalance is the bottleneck, continuous batching to reduce execution time variance—or separating prefill and decode into distinct pipelines—is a more direct fix than Zero-Bubble-style schedules.