MoE architectures became mainstream in production LLM serving between 2023 and 2024. When Mixtral 8x7B was released, the claim that "sparse activation reduces inference cost" spread quickly — and that claim is about half true. FLOPs go down. VRAM does not. That asymmetry is exactly what makes MoE serving cost calculations non-trivial.
Mixture of Tokens (MoT) approaches the problem from a different angle. The architecture was proposed in Google DeepMind's Soft MoE paper, and it eliminates the routing decision entirely. FLOPs actually increase, but the load imbalance and token dropping problems that stem from routing disappear. The two architectures are targeting different bottlenecks.
MoE's Sparse Activation and the Memory Paradox
MoE replaces the FFN layer in a Transformer with N experts, where each token passes through a router and selects only K of them. This is the core idea from Shazeer et al. 2017's Sparsely-Gated MoE; Mixtral 8x7B uses Top-2 routing, selecting 2 experts out of 8.
Only K/N of the total parameters are activated per token. According to Mistral AI's technical report, Mixtral 8x7B has 46.7B total parameters, of which 12.9B are active during inference. The FLOP reduction tracks that ratio — that part is correct.
VRAM is a different story. Because which expert each token selects is determined at runtime, all 8 expert weight matrices must reside in memory. Serving Mixtral 8x7B in BF16 requires roughly 87–93 GB — you need two A100 80GB GPUs. The fact that only 12.9B parameters are active does not translate to proportionally less VRAM.
| Dense model (7B class) | Mixtral 8x7B | |
|---|---|---|
| Total parameters | ~7B | 46.7B |
| Active parameters at inference | ~7B | 12.9B |
| BF16 VRAM requirement | ~14GB | ~90GB |
| FLOPs per token | baseline | ~2/8 of baseline |
It's light by FLOP count, but far heavier by memory footprint. That asymmetry is what makes MoE serving cost complicated.
Beyond memory, there's another issue. MoE sets an upper bound on how many tokens each expert can process — the expert capacity:
expert_capacity = (capacity_factor × total_tokens × top_k) / num_experts
When tokens within a batch concentrate on a particular expert, this buffer is exceeded and the overflow tokens skip that expert entirely. In production, if this drop rate exceeds 1–5%, output quality starts to degrade noticeably. The router also tends to amplify imbalance through self-reinforcement during training, making it easy for certain experts to consistently receive the majority of traffic as a kind of steady state.
Mixture of Tokens: Eliminating the Routing Decision Entirely
The key idea in Soft MoE — Mixture of Tokens, proposed by Puigcerver et al., ICLR 2024 — is to replace the hard assignment of tokens to experts with a soft weighted combination.
In MoE, tokens select experts. In MoT, each expert receives a different weighted mixture of all tokens in the input sequence. Each expert processes its own distinct "soft token" — a combination of the full sequence — and the outputs are merged back via a weighted sum to the original token positions. No token exclusively selects any particular expert.
This eliminates load imbalance: every expert processes the same amount of input, a mixture of the entire sequence. Token dropping is gone too, since there's no capacity buffer to overflow. And because there's no routing decision, there's no need for an auxiliary loss to force balance.
The numbers reported in the paper: a Soft MoE Huge/14 (128 experts, 16 MoE layers) with more than 40× the parameters of a ViT Huge/14 achieved substantially better image classification quality while adding only 2% to inference time. This is possible because the input size each expert processes is fixed regardless of the number of experts, so scaling up expert count doesn't proportionally increase per-expert compute.
How the Routing Strategy Changes the Serving Cost Structure
Here's how the two architectures compare from a serving perspective:
| MoE (Hard Routing) | MoT (Soft Routing) | |
|---|---|---|
| Active parameters per token | K/N | All N experts |
| VRAM requirement | All parameters resident | All parameters resident |
| FLOPs per token | Reduced (K/N ratio) | Increased (N×) |
| Load imbalance | Present (mitigated with auxiliary loss) | None |
| Token dropping | Possible | None |
| Routing overhead | Router compute + scatter/gather | Weighted-sum matrix ops |
| Batch efficiency | Improves with larger batches | Uniform regardless of batch size |
MoE's GPU utilization problem is most pronounced during the decode phase. When each token in a batch selects different experts, each expert only processes the tokens routed to it. From the FFN's perspective, the effective batch size drops to roughly K/N of the full batch. vLLM's expert parallelism documentation explicitly acknowledges this — even with EP enabled, the per-expert batch size shrinks during decoding, causing GPU utilization to drop.
This problem gets better as batch size grows. With a batch of 128, each Mixtral expert receives on average 32 tokens (128 × 2/8), which is enough for reasonable GPU utilization. With a batch size of 1, routing and scatter/gather overhead can exceed the cost of the actual matrix operations. This is why MoE's advantage shows up in throughput with large batches, not in single-request latency.
MoT doesn't have this structural problem. Every expert always processes the full input, so GPU utilization is uniform regardless of batch size. The trade-off is that FLOPs scale honestly with expert count — there's no hiding that cost.
What Production Systems Are Choosing Today
Mixtral 8x7B, DeepSeek-MoE, and Grok-1 all went with MoE. DeepSeek-MoE pushed the architecture further by splitting experts into finer-grained units and designating some as shared experts that all tokens pass through, reducing load imbalance while improving specialization. Both vLLM and TensorRT-LLM support MoE expert parallelism; vLLM v1 (early 2025) added FlashInfer integration and MoE kernel autotuning, automatically selecting the optimal configuration per batch size at startup.
The reason MoT is rarely used in production serving comes down to this structural difference. The primary motivation for choosing MoE is FLOP reduction — and MoT gives that up. The fact that the Soft MoE paper focuses its results on vision transformer experiments also means there isn't yet a clear answer on how expensive the full-sequence weighted-sum computation across all experts actually is at LLM scale.
MoE is the right choice when you have sufficiently large batches and can build out expert parallelism infrastructure. If routing instability is unacceptable, your FLOP budget has headroom, or training stability is the top priority in a research context, MoT-style architectures are worth considering. The two structures have different bottlenecks — MoE's are routing decisions and communication, MoT's is raw compute. Which bottleneck is more expensive depends on your hardware and workload.