MoE has already been production-validated through Mixtral, DeepSeek, and the Qwen model family. Mixture of Tokens (MoT), by contrast, is still an unfamiliar name. Both claim to reduce FLOPs compared to dense models — so what's the difference?
The distinction comes down to which axis sparsity is applied along. MoE operates on the parameter axis — deciding, per layer, which expert (FFN) each token is routed to. MoT operates on the sequence axis — deciding, per layer, which tokens in the sequence are actually computed. The sparsity directions of the two techniques are orthogonal, and they can be applied simultaneously within the same model.
MoE: More parameters, same FLOPs
The Mixtral 8x7B architecture illustrates this clearly. Total parameters are 46.7B, but each token only passes through 12.9B of them. Each of the 32 layers contains 8 expert FFNs, and for each token, the top-2 experts are selected and computed. Per-token FLOPs come to roughly 19% of a 70B dense model (13B / 70B), and inference is about 6× faster than Llama 2 70B.
MoE is not a technique for reducing parameters. The goal is to pack 46.7B worth of knowledge into the model while keeping active FLOPs at the level of a 12.9B dense model. "More parameters, fewer active FLOPs" — that's the entirety of what MoE does. Compared against a 12.9B dense model, MoE wins by a wide margin; compared against a 46.7B dense model, performance is similar or slightly worse, but FLOPs are substantially lower.
From a serving perspective, MoE introduces problems around expert capacity. There's a cap on how many tokens each expert can handle, and tokens that exceed the cap are either dropped or sent to an overflow expert. As batch size grows, load imbalance — where some experts receive far more tokens than others — becomes more pronounced. Adding a load balancing loss during training doesn't fully eliminate imbalance at inference time.
In expert-parallel settings, this translates directly into GPU utilization problems. Since different experts process different numbers of tokens, some GPUs are busy while others sit idle. This is exactly why vLLM introduced the Expert Parallel Load Balancer (EPLB). Larger batch sizes improve utilization, but KV cache memory constrains how large batches can be — forcing a tradeoff between the two.
MegaBlocks (Gale et al., 2022) approaches this problem differently. It reformulates MoE computation as block-sparse matrix operations, avoiding token dropping while still achieving efficient GPU kernel utilization. It reports up to 40% faster training than Tutel and 2.4× faster than Megatron-LM.
MoT: Skip computation at the token level
The idea behind MoT is straightforward. Not every token needs to pass through the Attention and FFN of every layer. In some layers, only the "important" tokens need full computation; the rest can skip that layer entirely.
The most prominent implementation of this approach is Mixture-of-Depths (MoD) (Google DeepMind, 2024). It fixes the number of tokens to process per layer at k, uses top-k routing to select which tokens get computed, and lets the unselected tokens bypass the layer entirely via the residual connection. The total FLOP budget is fixed in advance, but which tokens receive heavy computation at which layers is determined dynamically based on input content. Within the same FLOP budget, MoD matches baseline performance and is up to 50% faster at post-training sampling. The paper also experiments with a setting that reaches equivalent performance using 25% fewer training FLOPs.
An earlier, simpler approach is Token Merging (ToMe) (ICLR 2023). Applied to Vision Transformers, it merges similar tokens to reduce sequence length, achieving 2× throughput on ViT-L @ 512 with only a 0.2–0.3% accuracy drop. The motivation aligns with MoT for LLMs, but the mechanism differs (merging vs. skipping), and the target domain differs (images vs. text).
The critical difference between MoT and MoE lies in KV cache behavior. With MoE, every token passes through every layer regardless of which expert is chosen, so KV cache size grows in direct proportion to sequence length. With MoT, skipping a token at a given layer means the attention computation for that layer is skipped entirely — no KV cache entry is created at all. This is the structural reason MoT offers a more direct memory advantage than MoE for long-context serving.
Tradeoffs between the two approaches
| MoE | MoT (MoD family) | |
|---|---|---|
| Sparsity axis | Parameters (expert FFN) | Sequence (token skip) |
| FLOP reduction site | FFN | Attention + FFN |
| Total parameters | Increases (scales with expert count) | Unchanged |
| KV cache impact | None | Can directly reduce |
| Per-layer sequence length | Fixed | Variable |
| Source of GPU irregularity | Expert dispatch imbalance | Variable token count |
| Production maturity | High | Low (research stage) |
From a hardware efficiency standpoint, both techniques reduce GPU utilization, but for different reasons. MoE's irregularity comes from expert dispatch — tokens are dynamically routed to different GPUs, triggering all-to-all communication and causing per-expert token count imbalances. MoT's irregularity comes from variable sequence length — the number of active tokens per layer varies with each input, so token counts differ across layers and across samples within a batch.
EPLB and MegaBlocks are optimizations that target MoE's irregularity. PagedAttention can partially accommodate MoT's variable memory patterns, but if the number of active tokens varies per layer, the number of KV cache blocks needed also differs per layer — and predicting that count before prefill begins remains an open problem.
Using both together
Combining MoE and MoT multiplies the sparsity benefits. But having both expert dispatch irregularity and variable sequence length simultaneously means the batch scheduler must handle operations of different sizes running on different GPUs across different layers within a single forward pass. KV cache sizes vary both per layer and per input, making pre-allocation difficult even with PagedAttention. Token routing patterns also differ between prefill and decode, requiring asymmetric handling.
As of the current state of vLLM and SGLang, an optimized serving path for combined MoE + MoT models is not yet complete. The MoD family only recently moved past the 2024 paper stage, and examples of it being applied to large-scale LLM pretraining in practice are still rare. Reaching the same level of end-to-end production serving stack validation that MoE has achieved will require another step forward.