Tensor Parallel vs Pipeline Parallel: Choosing the Right LLM Inference Strategy

When you first serve a 70B model, the first question is "how many GPUs?" and the second is "how do you split it?" That second question is more involved than it looks. Even with 8 GPUs, whether you configure TP=8, TP=4×PP=2, or run two separate TP=4 instances moves TTFT and throughput in opposite directions.

Where a Single GPU Hits the Wall First

Loading Llama-3 70B in BF16 requires 140GB for weights alone. A single H100 (80GB) can't hold it, so two cards is the minimum. At BF16, the 405B weighs in at 810GB — even 8×H100 requires dropping to FP8 just to fit. Memory capacity is the first reason that forces you into distributed inference.

The second reason is different. Even when a model fits on a single GPU, you may still choose to distribute across multiple GPUs to improve throughput or reduce latency. These two objectives call for different distribution strategies. Trying to solve both memory constraints (model parallelism) and throughput scaling (data parallelism) with a single strategy makes it hard to even identify where you're losing efficiency.

Tensor Parallelism (TP): Splitting a Layer Horizontally

The core idea of tensor parallelism, formalized by Megatron-LM (Shoeybi et al., 2019), is to distribute a single Transformer layer across multiple GPUs for simultaneous computation. The QKV projection in attention and the first linear layer in the FFN are split column-parallel; the output projection and the second FFN linear layer are split row-parallel.

Column-parallel splits the weight matrix along the column dimension so each GPU independently computes a partial output. Row-parallel has each GPU compute a partial sum, then combine results via all-reduce. This gives you two all-reduces per layer — one each in the attention block and the FFN block.

That all-reduce is the central variable in TP. Under ring-all-reduce, the data moved per GPU is roughly 2 × hidden_size × bytes_per_element. For Llama-3 70B (hidden_size=8192, BF16=2 bytes), that's about 32KB per all-reduce, 64KB per layer, and roughly 5MB traversing the network per decode step across all 80 layers.

NVLink availability is the decisive factor here. H100 SXM's NVLink 4.0 delivers 900GB/s bidirectional per GPU. PCIe 5.0 x16 is 64GB/s unidirectional — a 14× bandwidth gap. Benchmarks on PCIe-only servers like the L40S show that running TP=4 causes 40–50% of inference time to disappear into communication. The vLLM documentation explicitly recommends PP over TP in environments without NVLink.

Efficiency Degrades as TP Scales Up

In theory, TP=N should yield an N× speedup. In practice:

The numbers published in vLLM GitHub issue #8089 for an 8×H100, Llama-3 70B FP8 setup make the problem even more stark:

TP=8 delivers 16% lower throughput and 35% higher TTFT than two TP=4 instances behind a load balancer. Per-token latency is nearly 4× worse. Once all-reduce latency exceeds computation time, adding more GPUs actually makes things slower. This is a case where the intuition that "TP=8 is always better on NVLink" simply doesn't hold.

Pipeline Parallelism (PP): Stacking Layers Vertically

PP divides Transformer layers into multiple stages, with each GPU responsible for a different subset of layers. Inter-GPU communication happens only at stage boundaries — a simple P2P transfer of activations from one stage to the next. There's no all-reduce, so PCIe connectivity can handle it without NVLink.

The problem is pipeline bubbles. With PP=4, a sequence must pass through four GPUs in order. While the first microbatch is being processed by stage 4, the stage 1 GPU sits idle. GPipe (Huang et al., 2019) addressed this by filling the idle time with multiple microbatches. The bubble fraction formula is:

bubble_fraction = (pp_stages - 1) / (pp_stages - 1 + num_microbatches)

With PP=4 and 8 microbatches, the bubble fraction is 3/(3+8) ≈ 27%. More microbatches reduce the bubble, but also mean larger effective batch sizes.

Why PP Is Particularly Costly During Decode

Decode generates tokens one at a time autoregressively. In practice, there's effectively only one microbatch per step. With PP=4 and microbatches=1:

3 / (3 + 1) = 75%

Three out of four GPUs are idle. To improve throughput, you need enough concurrency so that multiple requests are simultaneously filling the pipeline stages. In a real-time conversational setting with sparse, intermittent requests, PP can drop GPU utilization to 25%.

Prefill vs. Decode: Which Strategy Wins?

The two phases have fundamentally different compute characteristics.

Prefill processes the entire prompt at once. For a 1024-token input, the hidden state matrix is [1024 × 8192]. Both attention and FFN are dominated by large matrix multiplications, and the GPU is compute-bound. TP splits these large matrices across GPUs for parallel computation, directly reducing prefill latency. Even with non-trivial all-reduce overhead, sufficient NVLink bandwidth means the computation savings outweigh the communication cost.

Decode is different. Each step processes a hidden state of [batch_size × 8192], which at small batch sizes reduces to a [1 × 8192] vector-matrix operation repeated every layer. This is memory-bandwidth bound. The compute itself finishes quickly, but the all-reduce can take longer. Scaling TP increases all-reduce frequency and load, while the compute savings shrink. PP, with infrequent stage-boundary communication and each GPU handling its assigned layers exclusively, has a throughput advantage at high concurrency during decode.

This difference in arithmetic intensity between the two phases is the axis that determines your parallelism strategy. It's also the motivation behind prefill-decode disaggregation architectures. Physically separating the phases lets you independently tune each fleet — aggressive TP for prefill, higher PP ratios for decode with enough concurrency to absorb the bubbles.

DP and EP — When You Need a Separate Axis

Data parallelism (DP) replicates the model across multiple instances and distributes requests across replicas. It's the right choice when GPU memory isn't the constraint and you want to scale throughput further. The downside is that the KV cache isn't shared across replicas, so prefix cache hit rates fall inversely with replica count. In RAG pipelines that reuse long system prompts, blindly scaling DP spreads the cache benefit thin and can actually increase TTFT.

Expert parallelism (EP) is specific to MoE models. For architectures like DeepSeek-V3 or Mixtral that selectively activate experts per layer, EP distributes the experts themselves across GPUs. This introduces load imbalance when tokens concentrate on specific experts, plus all-to-all communication overhead.

Same 8 GPUs, Different TP×PP Configurations

Measured numbers across different configurations on a single 8×H100 node:

For Llama-3 70B FP8, a single TP=8 instance loses on throughput to two TP=4 instances. This is a real counterexample to the intuition that consolidating GPUs into one large parallel group is always better — even on NVLink.

Extremely large models like the 405B, which needs 810GB in BF16, are a different story. 8×H100 is the minimum unit and TP=8 is essentially mandatory. From there, your options are quantizing weights to FP8 to bring it down to ~405GB, or adding PP across multiple nodes to get more memory headroom.

Configuring TP and PP in vLLM:

# Single node, 8 GPUs, TP=8 single instance
vllm serve meta-llama/Llama-3-70b-instruct \
  --tensor-parallel-size 8

# Single node, two TP=4 instances (separate processes, behind a load balancer)
vllm serve meta-llama/Llama-3-70b-instruct \
  --tensor-parallel-size 4

# 2 nodes, TP=4 per node, PP=2 across nodes
vllm serve meta-llama/Llama-3-70b-instruct \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 2

For PP=2 and above, vLLM coordinates inter-node workers through Ray, and activation transfers at pipeline stage boundaries are handled via NCCL P2P communication.

Practical Decision Criteria

Organized by scenario:

• Model fits on a single GPU: If the goal is throughput, DP (multiple replicated instances) is the simplest and most effective approach. For lower latency, stop at TP=2. Efficiency drops to 70–85% at TP=4.

• NVLink servers (H100 SXM, etc.): Try scaling TP up to 8, but benchmark directly against two TP=4 instances. TP=8 losing is common.

• Non-NVLink servers (L40S, A10G, etc.): Cap TP at 2 and use PP for the rest. On PCIe, TP=4 and above lets communication consume half of inference time.

• Strict latency SLAs: Minimize PP. During decode, PP=4 can push the bubble fraction to 75%. Low concurrency combined with PP is the worst case.

• Throughput maximization: Scale up PP stages, but keep concurrency high enough to dilute the bubbles. If you can't fill the pipeline with enough microbatches, PP expansion just wastes GPUs.

• Multi-node scaling: Use NVLink for TP within a node and InfiniBand or Ethernet for PP across nodes. Spanning TP across node boundaries is practically infeasible due to the sharp bandwidth drop.

The phenomenon of TP=8 losing throughput to two TP=4 instances is observed in practice even on NVLink. The optimal configuration depends on concurrency and model size, so the only reliable method is to benchmark TTFT and throughput directly across different configurations.