Dense models are simple. One token comes in, and every parameter wakes up to participate in the computation. Running Llama 3 70B on a single request requires all 70B parameters resident in memory, and the entire set activates on every token. Inference cost scales directly with model size.
Sparse activation is an attempt to break that structure — load all the parameters, but only use a fraction of them for any given computation. Two concrete realizations of this idea are MoE (Mixture of Experts) and MoT (Mixture of Tokens). The names sound similar, and both are classified as sparse-activation LLMs, but the unit to which sparsity is applied is completely different. And that difference pushes the inference-time bottleneck in entirely different directions.
MoE — the theory and reality of expert routing
Mixtral 8x7B (Mistral AI, 2024) brought MoE into the mainstream. It replaces FFN layers with 8 experts, and a router selects the top-2 for each token, taking a weighted sum of their outputs. Of the 46.7B total parameters, only 12.9B are active at any time. The FLOP count per token is comparable to a dense 13B model.
In theory, fewer FLOPs should mean lower latency. In practice, two problems get in the way.
The first is load imbalance. Routers tend to converge during training toward favoring certain experts. Within a batch, one expert might receive 30% of tokens while another gets only 5%. Tokens that exceed an expert's capacity are either dropped or rerouted. Switch Transformer (Fedus et al., 2021) mitigated this with an auxiliary loss — adding ∑(f_i × P_i) as a loss term, where f_i is the fraction of tokens routed to expert i and P_i is the router probability, to encourage a uniform distribution. Even so, at inference time the router operates without this auxiliary loss, so the imbalance resurfaces.
The second is all-to-all communication. When the expert count grows beyond what fits on a single GPU, experts are distributed across multiple GPUs. In this expert-parallel setup, every token must be sent to the GPU hosting its assigned expert and the result returned — all-to-all communication is unavoidable. Empirical measurements show that all-to-all communication accounts for up to 79.2% of total inference time under synchronous expert parallelism, and the fraction grows with batch size. Halving the FLOPs doesn't translate proportionally to lower latency when communication overhead dominates.
The DeepSeek-V3 technical report attacks this from two angles. Despite an extreme configuration — 671B total, 37B active, top-8 out of 256 experts — it caps routing to at most 4 nodes per token, bounding the all-to-all communication scope. Load balancing drops the auxiliary loss in favor of a learnable per-expert bias added to the routing score. On H800s, this achieves prefill at 9,213 tokens/s and decode at 1,850 tokens/s.
MoT — mixing tokens instead of routing them
Mixture of Tokens (NeurIPS 2024) starts from a different premise. Rather than each token selecting an expert, each expert receives a weighted mixture of tokens from the batch as its input. A controller (an FC layer) uses continuous weights to decide how much of each token goes into each expert. The notion of a token being "assigned" to a specific expert doesn't exist.
Routing here is not discrete. Every expert sees every token — just with different weights. Because every expert always processes the same number of mixed inputs, load imbalance is structurally impossible. Routing decisions are differentiable, enabling end-to-end training without auxiliary losses, and training stability is on par with a vanilla Transformer.
The difference in the computation graph between MoE and MoT in one sentence: MoE makes the token→expert assignment sparse; MoT makes the expert→token mixture weights dense. In MoE, a token can be dropped and never reach a given expert; in MoT, no token is ever dropped and no expert ever sits idle.
The NeurIPS 2024 paper reports that MoT reaches the same loss as a vanilla Transformer at 33% of wall-clock time and 25% of FLOPs — a 3× wall-clock speedup and 4× FLOP reduction.
Inference cost comparison
| Dimension | MoE | MoT |
|---|---|---|
| Sparsity unit | Expert selection (discrete) | Token mixture weights (continuous) |
| Load imbalance | Structurally inherent | Structurally absent |
| Distributed inference communication | All-to-all (up to 79% of latency) | N/A |
| KV cache structure | Same as standard attention | Same as standard attention |
| Auxiliary loss | Required (or learnable bias) | Not needed |
| Framework support | vLLM/SGLang official support | None |
| Production models | Mixtral, DeepSeek-V3, and others | None |
KV cache behavior is identical for both architectures — neither touches the attention structure, so optimizations like paged attention and prefix caching apply equally to both. The only place where the two diverge from a serving-stack perspective is in FFN layer processing.
Deployment considerations
MoE is the easier choice to serve today. vLLM has official support for expert parallelism and lets you swap all-to-all backends — NCCL EP, DeepEP, and others — as plugins. There are already numerous production-validated models: Mixtral 8x7B, DeepSeek-V2/V3, the Qwen MoE family, and more.
MoT is a different story. As of the NeurIPS 2024 paper, no serving framework supports production deployment of MoT. The cross-example aggregation scheme requires changing how batches are constructed, which conflicts with the standard prefill/decode separation. There are no production deployments to point to.
Even if you go with MoE, skipping load-imbalance tuning will leave GPU utilization well below expectations. Severe expert imbalance means one GPU sits idle while it waits for others. Without adopting something like DeepSeek-V3's learnable bias approach, or adding explicit routing constraints at inference time, there will be a significant gap between theoretical and observed throughput.
MoT does structurally sidestep this problem — but how cross-example mixing interacts with large-scale autoregressive generation in practice, and how it integrates into a serving stack, remain open questions.