MoE vs MoT: Token Routing Architectures and Serving Costs

What's the Difference Between MoE and MoT — Token Routing and Its Serving Costs

In a dense model, every token activates every parameter. That means passing through a single Transformer FFN block requires multiplying the full weight matrix, and FLOPs scale linearly with parameter count. Sparse activation is an attempt to break that relationship: increase the number of parameters to raise learning capacity, while activating only a subset at inference time to keep FLOPs low. The tradeoff is that all parameters must reside in memory, and there's routing overhead to decide which parameters to activate.

Mixture of Experts (MoE) and Mixture of Tokens (MoT, also called Soft MoE) are two approaches to implementing sparse activation. Both aim at the same goal, but they differ in the unit of routing decision — selecting an expert versus mixing tokens. That single difference creates entirely different problems in serving systems.

MoE: Discrete Decisions Over Expert Selection

This has been the dominant architecture since Shazeer et al. (2017) introduced Sparsely-Gated MoE. Each Transformer layer splits its FFN block into N expert FFNs, and a per-token gate (router) decides which experts to use. Top-K gating is the standard. With K=2, the formulation is:

G(x) = softmax(TopK(x · W_g, K=2))
y = Σ_i  G_i(x) · FFN_i(x)

W_g is a learned gating weight matrix, and TopK keeps only the top-K scores, masking the rest to −∞. Token x passes through only the 2 selected experts, and their outputs are combined via a weighted sum.

Mixtral 8x7B makes this concrete. Total parameters are 46.7B, but only 12.9B are active per token. Inference FLOPs are roughly equivalent to a dense 13B model — yet the full 46.7B must be loaded into memory. That's 3.6× the VRAM of a dense 13B model, at the same throughput.

Load Imbalance: The Real Problem in MoE Serving

In theory, 8 experts should receive tokens uniformly. In practice, they don't. Even Shazeer et al. (2017) documented that without an auxiliary load balancing loss, some experts collapse and absorb most of the tokens. Even when this is suppressed during training, imbalance can re-emerge at inference time under certain input distributions.

When tokens in a batch concentrate on a particular expert, the other experts sit idle while that one expert processes requests back-to-back. Because GPUs handle matrix multiplications per expert, if one expert gets 32 tokens and another gets 2, the entire layer's latency is determined by the 32-token expert. This stretches the latency tail.

The problem naturally improves with larger batch sizes. With enough tokens, the variance in tokens-per-expert decreases. That's why MoE is more manageable for offline batch inference. In real-time serving, batches are small and input distributions can be skewed, so worst-case behavior appears frequently.

DeepSeek-MoE attacked this from a different angle. It uses fine-grained experts — splitting FFN experts into smaller units to increase expert count at the same compute budget, narrowing the knowledge scope per expert. Load balancing during training applies both expert-level and device-level losses. The follow-up, DeepSeek-V3, moved away from auxiliary losses entirely, instead dynamically adjusting router biases: if an expert is used more than average, its bias is lowered to reduce the probability of it being selected for subsequent tokens.

When Expert Parallelism Becomes Necessary

Loading a 46.7B model onto a single GPU is not feasible, so distributed inference is required. For MoE, this means Expert Parallelism (EP) on top of Tensor Parallelism (TP). EP places each expert on a different GPU. When a token is routed, it must be sent to the GPU hosting that expert, making all-to-all communication unavoidable.

vLLM now has official EP support, using a combination of --tensor-parallel-size and --data-parallel-size to determine EP size.

# 8-way expert parallelism on a single 8-GPU node
vllm serve deepseek-ai/DeepSeek-V3-0324 \
    --tensor-parallel-size 1 \
    --data-parallel-size 8 \
    --enable-expert-parallel

EP_SIZE is computed automatically as TP_SIZE × DP_SIZE. Adding --enable-eplb activates the Expert Parallel Load Balancer, which collects per-expert load statistics every forward pass and periodically redistributes experts across GPUs. See the vLLM EP documentation for full configuration options.

Soft MoE: Replacing Discrete Routing with Token Mixing

Puigcerver et al. (Google DeepMind, 2023) reframe the problem entirely. Instead of making a discrete decision about which expert to select, Soft MoE continuously mixes tokens and passes the blended inputs to each expert.

The core mechanism is a token-to-slot assignment matrix. Given m tokens, n experts, and s slots per expert:

Φ = softmax(X · Φ_W)            # (m × n·s) assignment matrix, column-wise softmax
X̃_i = Φ_i^T · X                # input to i-th expert: s slots, each a weighted sum over all tokens
Ỹ_i = FFN_i(X̃_i)               # expert computation
Y = Φ_i · Ỹ_i  (summed over i) # scatter outputs back to original token positions

Each expert's input is not a single token but a weighted average of all tokens. Every expert always receives exactly s inputs, so per-expert compute is constant regardless of how many tokens are in the batch.

Load imbalance disappears by construction, since every expert always processes the same number of slots. The paper reports that Soft MoE Huge/14 (128 experts, 16 MoE layers) has over 40× more parameters than ViT Huge/14 while adding only ~2% to inference time.

The Compatibility Problem with Autoregressive Generation

So why is Soft MoE absent from LLM serving stacks?

It breaks the token independence assumption. Since each expert input is a weighted sum of all tokens, the output at a given position depends on every other token in the sequence. In autoregressive generation, producing the current token only requires attending to previous tokens' KV values, which are cached and reused. In Soft MoE, adding a new token forces recomputation of the mixing weights across all tokens — the fundamental assumption behind KV caching breaks down.

The Soft MoE paper acknowledges this, and the experiments are conducted primarily on ViT-based image classification. Applying it to text generation requires resolving compatibility with causal masking, and no practical solution at the scale of large language models has emerged yet.

Serving Metrics Side by Side

Memory footprint is the same for both — all expert parameters must be loaded regardless of approach. Soft MoE's advantage over MoE is compute balance, and using that advantage for text generation still has unsolved problems.

The KV cache difference is particularly significant. In MoE, the KV cache only involves the attention layers; it has no interaction with the expert FFN layers. Regardless of which expert a token passes through, KV values accumulate the same way. vLLM's PagedAttention operates under this assumption and applies without modification to MoE models. In Soft MoE, token mixing happens across all FFN layers, so this assumption does not hold.

What to Check First When You Get an MoE Model

In practice, the models you'll actually be serving today — Mixtral, DeepSeek-V3, Qwen MoE, and similar — are all MoE. There's essentially no chance of encountering Soft MoE in production right now. The reason to understand Soft MoE isn't deployment — it's to clarify why load imbalance in MoE is such a persistent problem. Eliminating the discrete routing decision makes the problem disappear, and the reason you can't do that is the collision with autoregressive generation. Once you understand that, the direction for MoE serving optimizations becomes much clearer.

When you receive an MoE model, start by checking three things. Calculate the theoretical active parameter ratio from the expert count and Top-K value. Examine the expert-to-GPU assignment to determine whether an EP configuration is needed. If your batch size target is small (real-time, low-latency serving), budget latency separately under the assumption of worst-case imbalance. If you're serving with vLLM, enabling --enable-eplb for dynamic load balancing is increasingly becoming the default.

Research on MoT continues. There are ongoing attempts to work around the causal masking compatibility issue, and in architectures without the autoregressive assumption — such as diffusion-based language models — Soft MoE-style approaches may be a more natural fit. Whether and when that changes the landscape remains an open question.