In a dense model, even one with 70B parameters, every single token activates the full 70B during inference. That's the root cause of GPU costs in LLM serving. Sparse activation strategies emerged from a simple question: can we achieve the same quality while reducing the active parameters per token?
MoE (Mixture of Experts) is already a deployed answer. Mixtral-8x7B, DeepSeek-V2, and Grok-1 all use this architecture. Recently, "Mixture of Tokens (MoT)" has started appearing in the literature. It looks like a variant of MoE, but the routing direction is inverted — which means different bottlenecks show up at inference time. Putting the two strategies side by side from a serving cost perspective makes it clear where they align and where they diverge.
MoE: Tokens Choose Experts
The core structure of MoE replaces the FFN block in each layer with E independent experts, and a router selects only the top-k of them for each token. In Mixtral-8x7B, each layer has 8 experts and k=2 — every token always uses exactly 2. The total parameter count is 46.7B, but active parameters during inference are 12.9B, achieving performance comparable to Llama 2 70B at roughly 19% of the FLOPs.
Three problems compound at inference time with this architecture.
Expert load imbalance. Tokens in a batch should distribute evenly across 8 experts, but certain experts tend to attract disproportionate traffic during training. An auxiliary loss is added to prevent this, which itself constrains optimal routing — a real tradeoff. DeepSeek-V2 goes further, using three separate auxiliary losses (expert-level, device-level, and communication-level), with a fine-grained design that activates 6 out of 162 experts (160 routed + 2 shared).
Low GPU utilization. If a batch contains 100 tokens, each can select a different expert combination. Some experts might receive 20 tokens while others get only 3. With just 3 tokens, the matrix operations for that expert have too small a batch dimension to effectively utilize memory bandwidth.
All-to-all communication. With expert parallelism across multiple GPUs, tokens must be dispatched to the GPU holding the target expert and returned after processing — once per layer. vLLM's introduction of Wide Expert Parallelism (Wide-EP) achieved a 1.8x improvement in per-GPU throughput as of late 2025. Ironically, that number reflects just how much of a bottleneck all-to-all communication had been.
Mixture of Tokens: What Changes When Routing Is Inverted
MoT starts from a simple inversion: instead of tokens choosing experts, experts choose which tokens to process.
The early form of this idea is Expert Choice Routing by Zhou et al. (2022). Each expert selects the top-c tokens from the full batch that it wants to process. Since every expert handles exactly c tokens, load imbalance is structurally impossible — perfect load balancing with no auxiliary loss required. The paper showed over 2x faster training convergence compared to Switch Transformer (top-1 routing) and GShard (top-2 routing).
Mixture of Tokens (arXiv:2310.15961, 2023) takes this further. Instead of discrete token selection, it uses continuous aggregation. Each expert computes softmax weights over all tokens in the batch, constructs its input as a weighted sum of those token representations, processes it, and distributes the result back across tokens. Rather than a single token going wholesale to one expert, each expert receives a blend of information from multiple tokens.
This inversion is what fundamentally changes the compute pattern.
In MoE, the per-expert token count is determined at runtime — you can't know each expert's workload until the router produces its output. In MoT, the number of tokens each expert processes (c) is fixed upfront. That fixed shape is an advantage for GPU kernel optimization: the matrix operation dimensions are predictable ahead of time.
But a new constraint appears: KV cache access patterns. In autoregressive decoding, the KV cache stores keys and values from previous tokens. In MoT, the input to each expert is a weighted mixture of multiple token representations, which makes KV cache management for these "blended" representations non-trivial. If the batch composition changes during generation, the token mix seen by each expert changes accordingly. Optimizations like prefix caching and PagedAttention are designed around per-token identity — integrating them with MoT requires a separate design effort.
Where Inference Costs Diverge
On FLOP count alone, the two approaches look similar. Whether you use top-2 out of 8 experts or have each expert process 25% of all tokens, the amount of FFN computation executed is comparable. But real latency and throughput are determined by factors beyond FLOPs.
| MoE (token choice) | MoT (expert choice / soft) | |
|---|---|---|
| Load balancing | Requires auxiliary loss | Structurally guaranteed |
| GPU kernel shape | Uneven token count per expert | Fixed token count per expert |
| All-to-all communication | Per-layer with expert parallelism | Same structure, mixed representations |
| KV cache compatibility | Standard PagedAttention works | Non-trivial during decoding |
| Serving stack support | Mature (vLLM, TGI, etc.) | Early stage |
How painful all-to-all actually is in multi-GPU deployments depends on scale. In vLLM's large-scale DeepSeek serving experiments with 8-GPU expert parallelism, 2.2k tokens/sec per H200 was achieved — a result that required FlashInfer integration and automated kernel tuning.
In low-latency scenarios with small batch sizes, MoE's imbalance problem is more pronounced. Fewer tokens means higher variance in per-expert token counts and lower GPU utilization. MoT's fixed token count guarantee is theoretically advantageous here, but there are no published measurements of actual soft-mixing latency at small batch sizes. "Same FLOPs means same efficiency" is wrong for both approaches — memory access patterns and communication costs move independently of FLOP count.
The Serving Stack Still Favors MoE
vLLM has first-class support for expert parallelism and can serve Mixtral-8x7B, Mixtral-8x22B, and the DeepSeek-V2/V3 series in production. FP8 quantization on Mixtral-8x7B has shown a 25–30% throughput improvement over FP16 on H100.
As of 2026, there is no dedicated serving framework supporting MoT models. Training code is publicly available, but no serving stack exists with continuous batching, paged KV cache, and quantization. This gap puts MoT at a concrete disadvantage in deployment decisions. Even if its load balancing is cleaner and it may offer better GPU kernel efficiency, a missing serving infrastructure makes production deployment effectively impossible.
For MoT's routing ideas to translate into real serving gains, an implementation that solves the decoding-phase KV cache problem needs to appear. Until that happens, "the bottleneck is theoretically in a different place" is as far as the story goes.