The two names are structurally symmetric. Where MoE (Mixture of Experts) chooses which expert to send a token to, MoT (Mixture of Tokens) chooses which tokens deserve more computation. Both approaches share the goal of not using all parameters all the time, but the axis of sparsity is inverted — experts (parameter sets) in one case, tokens (compute depth) in the other — and that difference fundamentally changes where serving costs accumulate.
MoE: What Happens When Sparsity Lives in Expert Selection
Mixtral 8×7B is the de facto reference point for modern MoE architectures. Each of its 32 layers contains 8 expert FFNs, and each token passes through only 2 of them per layer via top-2 routing. Total parameters are 46.7B, but only 12.9B are active when processing any given token. Inference FLOPs come out to roughly 19% of Llama 2 70B. The clear value proposition of MoE is preserving the representational capacity of a large parameter count while dramatically cutting compute.
From a serving infrastructure perspective, costs arise in two places.
The first is routing imbalance. When tokens pile up on certain experts, you get a straggler problem: some GPUs are still churning through an overloaded expert while others are already waiting on the next batch. vLLM addresses this with EPLB (Expert Placement with Load Balancing), which collects load statistics each forward pass and periodically redistributes experts. Different domains — code, math, natural language — tend to favor different experts, so traffic skewed toward a particular domain tends to make imbalance worse.
The second is AllToAll communication. When experts are distributed across multiple GPUs, routing each token to the right expert requires an all-to-all collective. This works efficiently when tokens are evenly spread across a batch, but the moment skew appears, communication overhead and GPU idle time rise together. Approaches like Wide Expert Parallelism — spreading experts across 16–72 GPUs beyond a single NVLink domain (8 GPUs) — are attempting to address this problem at a different scale.
What Is Mixture of Tokens?
The Mixture of Tokens paper (arXiv:2310.15961) directly targets the discontinuity problem in MoE's hard routing. The idea is simple: instead of drawing hard boundaries like "token A goes to expert 1, token B goes to expert 2," each expert receives and processes a weighted combination of multiple tokens from the batch. The discrete top-k selection disappears, replaced by a continuous distribution of tokens across experts. Load imbalance is structurally eliminated — no token is exclusively assigned to any single expert. The paper reports a 3× training speedup over dense transformers, with quality on par with hard-routing MoE.
Google DeepMind's Mixture-of-Depths (MoD, arXiv:2404.02258) pushes this direction further toward dynamically assigning compute depth per token. Only the top-k tokens per layer actually run attention and MLP; the rest skip that layer via the residual connection. Under isoFLOP conditions, the paper reports up to a 1.5% performance improvement over a vanilla transformer, and claims that cutting per-forward-pass FLOPs by more than half still preserves baseline performance. Sampling speed improves by up to 50%.
The conceptual ancestor of this line of work is Graves's Adaptive Computation Time (ACT, 2016), which first concretized the idea of allocating a variable number of processing steps per input — in an RNN. That idea carried through Universal Transformer and eventually into the MoD/MoT family.
How Does a Different Sparsity Axis Change Serving Costs?
| MoE (Mixtral 8×7B) | MoT / MoD | |
|---|---|---|
| Sparsity unit | Expert (parameter set) | Token (compute depth / layer) |
| Parameter memory | Full expert weights (46.7B) | Same as a dense model |
| Active FLOPs | Low and predictable (~12.9B active) | Low on average, but unknown until runtime |
| Batching challenge | Expert load imbalance, AllToAll | Uneven per-token compute |
| FLOPs determined at | Routing decision (layer entry) | After forward pass completes |
| Cross-expert communication | AllToAll required | None |
| Serving maturity | vLLM, SGLang production-ready | Research stage |
The key serving difference is when FLOPs are determined.
In MoE, the routing decision is made at the start of each layer, and FLOPs are fixed at that moment. You can know ahead of time how much compute a request will consume in total, which is why MoE fits reasonably well with continuous batching.
In MoD, you don't know which tokens will actually pass through which layers until the forward pass runs. When requests in a batch traverse different layers at different points, it becomes impossible to group them by a consistent depth. Padding handles the mismatch, but tokens that took a shallow path burn empty compute. In a 32-layer model where half the tokens in a batch only use 16 layers, aligning the batch to 32 layers wastes roughly 25% of GPU compute to padding. Avoiding that waste requires grouping tokens with similar layer depths together — which introduces a new latency-throughput tradeoff.
Which Approach Fits Which Situation?
Framing this as "which one is better" is the wrong question. The two approaches serve different goals: scaling up parameter count versus reducing the compute budget within a fixed parameter count.
If you want to increase parameter count while holding FLOPs roughly constant, MoE is the right tool. The tradeoff is explicit: 46.7B parameters worth of representational capacity at roughly 13B-dense compute. The cost is that your serving stack needs to support expert parallelism, load balancing, and AllToAll communication optimization. vLLM and SGLang already provide this.
If you want to cut the compute budget within the same parameter size, MoT/MoD is the right direction. There's no cross-expert communication overhead, and the architecture isn't radically different from a dense model — so it looks easy to integrate into existing serving stacks. In practice, though, compute being determined at runtime makes latency SLAs hard to guarantee. Production deployments of MoD for P99 latency-sensitive workloads are rare compared to MoE, and scheduler approaches like "continuous depth batching" are still being explored, not production-validated.
Research into combining both approaches — dynamically determining both expert assignment and layer depth per token — is ongoing. The optimization variables double, and so does the serving complexity. Whether the performance gains justify that complexity remains an open question; the data isn't there yet.