MoE vs MoT: How Routing Granularity Reshapes LLM Serving

Sparse Mixture of Experts (MoE) is the dominant LLM architecture in production today. Mixtral 8x7B, DeepSeek-V3, and Qwen2-57B-A14B all fall into this category. The premise is straightforward: more parameters, fewer activations — expand model capacity while theoretically saving FLOPs. In practice, though, serving costs rarely drop as much as the theory suggests. The reason lies in the routing itself.

Mixture of Tokens (MoT) redesigns this structure from the opposite direction. The reference paper is Mixture of Tokens: Continuous MoE through Cross-Example Aggregation, posted to arXiv in late 2023 and presented at NeurIPS 2024. The difference between MoE and MoT is that the subject and direction of routing are inverted — and that inversion changes the entire cost structure of serving.

MoE's Basic Structure and the True Cost of Inference

Replace the FFN block in a standard transformer with N expert FFNs, and for each token select top-k of them to execute. Mixtral 8x7B runs 8 experts per layer with top-2 selection per token. Total parameters sit at 46.7B, but the parameters actually active in a single forward pass are around 12.9B.

Routing works by having a gating network (a linear layer followed by softmax) compute per-expert scores for each token and select the k highest-scoring ones. That simplicity produces three real costs.

Expert load imbalance. The gating network's tendency to favor certain experts shows up early in training and is hard to correct once it sets in. In Mixtral 8x7B, the token-count disparity between experts reaches as high as 25%. More concretely, tokens pile up on a small number of experts in the early and final layers, while the middle layers are relatively balanced. The imbalance worsens at smaller batch sizes — with too few tokens, a statistically uniform distribution is unlikely, and at batch size 1 it's common for nearly all tokens to land on one or two experts.

All-to-all communication. In distributed serving with expert parallelism, each GPU owns a different set of experts. Because the gating result isn't known until after it's computed, every GPU must send its tokens to whichever GPU holds the target expert, then receive the results — two all-to-all collectives per layer. Even within an NVLink domain this consumes roughly 20% of total execution time, and scaling EP from a single node to eight nodes drives the all-to-all fraction from 22% to 78%. Models with more experts hit this overhead ceiling sooner.

KV cache fragmentation. Even within a single prompt, different expert combinations are selected at each layer, making it complex to track which expert's context a given KV cache entry was generated under. When expert selection patterns diverge between prefill and decode, cache reuse drops.

These three costs eat into the theoretical FLOPs savings. That's why the claim "only 1/4 of parameters are active, so it's 4× faster" doesn't hold up in real serving.

What Is Mixture of Tokens — Inverting the Direction of Routing

MoT flips this structure around. Instead of tokens choosing experts, each expert soft-attends over the entire token set.

Each expert computes attention-like weights over all tokens in the batch (a token pool mixed cross-example) and aggregates a representation from them. There is no step where a specific token is routed to a specific expert. From the expert's perspective, it smoothly reads the entire token pool and extracts the information it needs directly. In the paper's terminology, this is "cross-example aggregation" — each expert processes a mixture of tokens drawn from different examples in the batch.

The immediate consequence of this inversion is the structural elimination of the load balance problem. Expert overloading simply doesn't exist. Computation is divided exactly across the number of experts, and auxiliary load-balancing losses are unnecessary. Training stability improves as well — MoE's discrete top-k selection stops gradients from flowing to unselected experts, leading to dead experts late in training, whereas MoT's soft access ensures every expert receives gradients at every step.

On performance: the paper reports 3× faster training than a dense transformer at equal FLOPs, with parity against state-of-the-art MoE architectures. Compatibility with autoregressive generation is also established — prior soft/continuous MoE variants either couldn't support autoregressive inference or suffered large performance degradations, and this paper is the first to resolve that.

New costs do appear, however. Because each expert must see all tokens in the batch, memory access and compute scale super-linearly as sequence length and batch size grow. In MoE, a token only visits its assigned expert; in MoT, every expert references the full token pool. That access cost maps directly to a memory bandwidth bottleneck.

Where the Two Architectures Diverge at Inference

The bottlenecks these two architectures hit in real serving occur at different points.

MoE is communication-bound. All-to-all is mandatory under expert parallelism, and that latency is tied directly to inter-GPU interconnect speed. The moment traffic crosses InfiniBand, latency spikes sharply. More experts (DeepSeek-V3 uses 256) means greater routing complexity and higher communication frequency.

When serving MoE models in vLLM, combining EP (expert parallelism) with TP (tensor parallelism) produces a meaningful difference over TP alone. Measured on H100 with Qwen3.5-35B-A3B (source):

That's roughly 33% higher throughput and 32% lower latency. The command to enable EP in vLLM:

vllm serve Qwen/Qwen3.5-35B-A3B \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --gpu-memory-utilization 0.9 \
  --max-model-len 32768

MoT is memory-bandwidth-bound. Each expert iterates over all tokens in the batch, so memory access complexity scales as O(B × L × E) with batch size B, sequence length L, and expert count E. As batches or sequences grow, the memory bandwidth ceiling is reached faster than linearly. There's no all-to-all communication, so on a single node MoT is free of communication latency — but that advantage erodes quickly as sequence length increases.

At large batch sizes and long sequences, MoT's memory access costs grow sharply. In distributed multi-node environments, MoE's all-to-all overhead becomes dominant. Which bottleneck is worse depends on the specifics of the serving environment.

Current State and Practical Choice

Virtually every architecture in large-scale production serving today is MoE. Major serving frameworks like vLLM and TGI have accumulated years of investment in expert-parallelism-based MoE optimizations. MoT variants remain in the research stage and have no first-class support in these frameworks.

The reason is structural. MoT assumes that tokens from different requests are mixed within the batch, which conflicts with the "per-request isolation" paradigm that current serving frameworks take for granted. The baseline assumption in serving is that each user request is processed independently; MoT requires experts to see tokens from other requests in the same batch. Serving this safely requires redesigning batch composition strategy, memory management, and scheduling logic from scratch.

From a serving engineer's perspective, MoE is the practical choice right now. That said, the problems MoT structurally eliminates — load imbalance, dead experts, auxiliary loss tuning — are genuine headaches in MoE deployments. The solutions just haven't made it to production yet.