What's Different Between MoE and MoT — How the Routing Granularity Changes the Serving Architecture
The first numbers that come up when studying Mixture of Experts (MoE) are usually Mixtral 8x7B's. Total parameters: 47B. Activated parameters per token: ~13B. Each layer has 8 FFN experts, and only the top-2 are selected, so the compute breaks down as attention (~5B) + 2 selected FFN experts (~2×4B) ≈ 13B. That's roughly the same FLOPs as a dense 13B model, at quality comparable to Llama 2 70B.
This architecture reduces FLOPs, but memory requirements stay the same — all 47B parameters must reside on GPU. At bfloat16 that's ~94GB, which won't fit on a single A100 80GB. That's exactly why sparse model serving is both attractive and painful.
Mixture of Tokens (MoT) starts from the same sparsity idea but inverts the routing granularity. In MoE, a gate network tells each token which expert to go to. In MoT, each token receives a soft-weighted mixture of multiple experts' outputs. That difference in routing granularity produces an entirely different set of problems at the serving layer.
MoE: Expert Capacity, Token Dropping, and the Cost of Distributed Serving
The central constraint of MoE routing is the expert capacity formalized by Switch Transformer (Fedus et al., 2021):
expert capacity = (T / N) × capacity_factor
T is the total number of tokens in the batch, N is the number of experts, and capacity_factor is typically between 1.0 and 1.25. Each expert accepts tokens only up to this capacity; any overflow skips that expert entirely. That's token dropping.
Token dropping is not a bug — it's a deliberate capacity trade-off. A higher capacity_factor reduces drops at the cost of wasted memory and compute; a lower one increases drops and degrades quality. In a well-tuned model, the token drop rate stays below 1% on the training distribution. When the batch composition at serving time diverges from that distribution, load concentrates on certain experts and drop rates spike. The problem almost never surfaces at batch size 1, but grows non-deterministically as batch size increases. Ignoring this at the serving layer means quality degradation that varies unpredictably with batch size.
Load imbalance follows directly from this. When tokens in a batch are semantically similar — say, a single-domain query flood hitting one batch — they concentrate on the same experts while the rest sit idle. An auxiliary load-balancing loss during training encourages even distribution, but that's a training-time incentive, not a serving-time guarantee. When the input domain shifts, imbalance reappears.
Distributed serving also changes the communication pattern. Tensor parallelism for dense models uses an all-reduce pattern — each GPU handles a slice of the layer and the results are summed. Because MoE routing dynamically determines which token goes to which GPU's expert, it requires all-to-all communication: all tokens in the batch are dispatched to the GPU holding the appropriate expert, processed, and returned. The communication volume scales with batch size × sequence length × hidden dimension, making expert placement a first-class variable in serving latency.
vLLM supports MoE-specific expert parallelism via the --enable-expert-parallel flag. Unlike the default tensor parallelism, which splits layers column/row-wise, expert parallelism distributes entire experts across GPUs. A single-node example:
vllm serve mistralai/Mixtral-8x7B-v0.1 \
--tensor-parallel-size 1 \
--data-parallel-size 8 \
--enable-expert-parallel
In TensorRT-LLM, expert_parallel_size can be set independently of tensor_parallel_size for finer-grained control.
Mixture of Tokens — What Soft Routing Changes
The MoT approach was formalized in Kim et al. (2023). The core idea is cross-example aggregation: tokens from multiple examples in a batch are sent to the experts, and each expert's output is returned to the originating token as a weighted sum. Because tokens receive a mixture of expert outputs rather than being assigned to a single expert, token dropping cannot occur by construction. The paper reported a 3× training speedup over a dense Transformer.
An earlier work in the same direction is Zuo et al.'s THOR (ICLR 2022). THOR eliminates the gating network entirely and uses random routing at both training and inference time, adding a consistency regularization loss to keep each expert's predictions coherent. They reported exceeding Switch Transformer by 2 BLEU while achieving comparable performance to top-tier MoE models with an 18× smaller model. The key takeaway from that work is that they found no evidence that gating-based routing outperforms random routing — a direct challenge to a foundational assumption in MoE research.
MoT's cost structure differs from MoE's. Soft routing keeps more expert computation active, so FLOPs savings can be smaller than with hard routing. Where a top-2-of-8 MoE activates only 25% of the FFN, MoT processes a higher fraction depending on the aggregation scheme. This also affects KV cache reuse. vLLM's prefix caching reuses the KV cache for identical prefixes, but with soft routing the expert mixture weights for the same token can vary with batch composition, causing more frequent cache invalidation.
Direct Serving Comparison
| Dimension | MoE (hard routing) | MoT (soft routing) |
|---|---|---|
| Token dropping | Occurs when capacity is exceeded | Structurally absent |
| Load imbalance | Emerges with skewed batch distributions | Even load across experts |
| Activated fraction | top-k / N (e.g., 25%) | Higher (implementation-dependent) |
| GPU communication pattern | All-to-all | All-reduce (closer to dense) |
| Batch size sensitivity | Larger batches ease imbalance | Relatively low |
| KV prefix caching | Supported | Difficult |
| Production serving stack | Supported (vLLM, TensorRT-LLM) | Not supported |
Under continuous batching, when a new request joins the batch the two architectures behave differently. In MoE, the new request's tokens shift the in-batch distribution, potentially adding load to specific experts. If capacity buffer headroom is insufficient, existing request tokens start dropping. MoT avoids this problem, but its total expert compute scales proportionally with the number of active requests.
The reason MoE throughput increases with batch size is that larger batches bring more token diversity, which evens out expert utilization. TTFT (Time to First Token), on the other hand, grows with batch size because more all-to-all communication occurs during the prefill phase.
Production Adoption
MoE is the current default architecture for large model serving. Mixtral 8x7B, DeepSeek-V2, and Switch Transformer all use hard routing. DeepSeek-V2 adds fine-grained expert segmentation (subdividing each expert further to increase specialization) and shared experts (always-active common experts), and uses device-limited routing to cap the number of target GPUs per token at M, bounding the all-to-all communication volume.
MoT remains in the research phase with no official support in major serving stacks. Figuring out how soft routing can coexist with autoregressive KV cache reuse is the outstanding engineering problem that stands between MoT and production deployment.