MoE (Mixture of Experts) has a way of surprising engineers who try to serve it. The active parameter count sounds small, but GPU memory fills up fast, and throughput collapses much faster than expected as batch size shrinks. That gap comes directly from the MoE routing structure itself.
MoE Fundamentals — Why Inference Is Expensive
The core idea of MoE is to replicate the FFN (Feed-Forward Network) layer N times and route each token through the top-k of those experts. Mixtral 8x7B always selects 2 of its 8 experts. Active parameters: 12.9B. Total parameters: 46.7B. In terms of compute, it's comparable to a 13B dense model.
Here's the catch: compute goes down, but memory footprint does not. Because you can't know in advance which token will pick which expert, all 8 experts' parameters must reside in GPU memory at all times. Serving Mixtral 8x7B in FP16 requires roughly 90 GB — more than a single H100 (80 GB) can hold.
Even harder to deal with than memory is load imbalance. When the router selects top-k experts per token, it naturally tends to concentrate traffic on certain experts. Training counteracts this with an auxiliary load balancing loss. Switch Transformer (Fedus et al., 2022) established this approach: it adds to the loss a sum of products between each expert's token-fraction f_i and its router probability P_i, penalizing skew. It helps, but doesn't fully solve the problem. In serving environments, request lengths vary and batch composition differs from training, so imbalance resurfaces.
Distributed serving adds communication overhead on top of that. With expert parallelism — spreading experts across multiple GPUs — each token triggers GPU-to-GPU all-to-all communication depending on which GPU holds its chosen expert. Every MoE layer requires two all-to-all passes: dispatch and gather. As batch size grows, this communication has been reported to account for up to 79.2% of total inference time.
Mixture of Tokens — What Changes When You Flip the Routing Direction
MoT reverses who does the routing. Instead of "which expert does each token go to," the question becomes "which tokens does each expert choose to process." This is also called Expert Choice routing, and was formalized in Zhou et al.'s "Mixture-of-Experts with Expert Choice Routing" at NeurIPS 2022.
The mechanics: given N experts and T tokens in the batch, each expert selects the top-c tokens by router score (c = k × T / N, where k is the average number of experts per token). Everything else mirrors MoE — the only change is that selection authority moves from tokens to experts.
This inversion structurally eliminates load imbalance. Each expert is designed to process exactly c tokens, so there's no way for load to pile up on a particular expert. The auxiliary load balancing loss is no longer needed — the penalty term that MoE had to bake into the training loss at the cost of muddying it simply doesn't exist here.
But this structure carries its own costs. The number of experts handling each token is no longer uniform. Some tokens get selected by multiple experts; others may not be selected by any. During prefill, this variability is fine. During decode, tokens are generated one at a time, and you can't know how many experts will select a given token before seeing the full batch — expert selection results only emerge once the entire batch is assembled. Serving MoT as-is for autoregressive decoding therefore requires a separate implementation strategy.
There's also a difference in KV cache behavior. In MoE, each token follows a fixed routing path through specific experts, which keeps cache management relatively straightforward. In MoT, the set of tokens each expert selects changes from batch to batch, making cache reuse patterns considerably more complex.
Serving Trade-offs
| Dimension | MoE (Token Choice) | MoT (Expert Choice) |
|---|---|---|
| Load balance | Requires auxiliary loss; runtime imbalance persists | Structurally perfect balance |
| Token processing guarantee | Tokens can be dropped if they exceed top-k capacity | Experts select tokens — no drops |
| GPU memory | All experts must reside in memory | Same |
| All-to-All communication | Uneven token destinations, unpredictable | Fixed per-expert token count, predictable |
| Decode serving | Mature support in vLLM, SGLang, etc. | Complex to implement; ecosystem support immature |
| Per-token compute | Uniform (k × FFN cost) | Variable |
High-concurrency, large-batch workloads: In MoE, the share of latency consumed by all-to-all communication grows steeply with batch size. In MoT, the per-expert token count c scales proportionally with batch size, making communication volume predictable and — in theory — giving it better throughput scalability.
Small-batch, many short requests: This is where MoE's compute savings are most fragile. With few tokens in the batch, most experts sit idle, and the fixed cost of keeping all expert parameters in memory erodes the savings. MoT's per-expert token count scales with batch size, so utilization holds up better in small-batch scenarios.
Long-context prefill: This is where MoT's batch-level load balancing shines most. Processing thousands of tokens at once, expert utilization stays uniformly high. The caveat is that no mainstream serving framework currently supports MoT in the decode phase, which is a hard constraint for real deployments.
Measured Numbers and Current State
Mixtral of Experts (Jiang et al., 2024) uses top-2 routing to activate only 12.9B of its 46.7B parameters, but serving it still requires 90 GB in FP16. It's a concrete demonstration that the promise of "small FLOPs" doesn't carry over to memory cost. With expert parallelism, all-to-all overhead easily exceeds half of end-to-end latency at large batch sizes.
On the Expert Choice side, Zhou et al. reported that, under the same FLOPs budget, their approach achieves lower perplexity and higher downstream accuracy than token-choice MoE. The gains come from eliminating dead experts and ensuring all experts train uniformly.
Today, vLLM and SGLang both support MoE expert parallelism; SGLang combines EP with grouped GEMM optimizations to reduce latency. Serving an MoT (Expert Choice) model in the decode phase natively is not yet available in any mainstream framework. The idea was validated at NeurIPS 2022, but serving-level implementation hasn't caught up. If you want to use MoT today, the realistic options are to implement decode support yourself, or restrict its use to prefill-only workloads such as document embedding or reranking.