Both approaches start from the same premise: not every parameter needs to run on every token. But the way they implement this is opposite. In MoE, tokens choose their experts; in Mixture of Tokens, the layer chooses which tokens to process. That single reversal in routing direction creates entirely different problems for the serving system.
The Routing Direction Is Inverted
In MoE, the layer structure stays the same across the board. Instead, a gating network decides which expert each token goes to. In Mixtral-8x7B, each layer has 8 experts (FFN blocks), and each token selects 2 via top-2 routing, with outputs combined as a weighted sum. Even tokens within the same sequence can be routed to different experts.
Mixture of Tokens works in the opposite direction. The layer decides which tokens it will process. The MoT proposed in Antoniak et al.'s NeurIPS 2024 paper uses cross-example aggregation, where a layer mixes tokens from multiple examples in the batch. A given layer might receive tokens A and B from the current sequence alongside token C from a different sequence, and process them together.
Whether the routing subject is the token or the layer changes everything: memory requirements, batch composition, and inter-GPU communication patterns.
MoE Architecture: The Real Serving Cost of Sparse Activation
Mixtral-8x7B has 47B total parameters. When processing a single token, only 13B of those parameters are actually activated. In theory, this should reduce compute relative to a dense model.
The problem is serving. At inference time, the remaining 34B parameters still have to live in GPU memory, because there is no way to know before runtime which token will route to which expert. Serving Mixtral-8x7B requires roughly 94 GB of GPU memory in fp16.
In multi-GPU setups, expert parallelism distributes experts across GPUs. vLLM supports this with five communication backends, including allgather_reducescatter, deepep_high_throughput, and deepep_low_latency. Each layer requires an all-to-all communication step: tokens are dispatched to the GPU responsible for their expert, and outputs are gathered back. On H100/H200 hardware, NVLink's 450 GB/s unidirectional bandwidth can absorb this cost. In PCIe environments, bandwidth is roughly one-tenth of that, and all-to-all immediately becomes the bottleneck. The vLLM forums report that on consumer GPUs without P2P connectivity, the implementation falls back to a naive broadcast, inflating unnecessary data transfer.
The more fundamental issue is load imbalance. When the gating network concentrates tokens on certain experts, only those GPUs stay busy. Recent research shows that an average of 18.6% of GPU time per MoE layer is wasted due to imbalance, with severe cases where GPUs are idle more than 80% of the time. MegaBlocks mitigates this with block-sparse operations, achieving 1.8–2.4× throughput over conventional frameworks, but it does not eliminate token skew at the source.
It is also worth noting that in Mixtral's experiments, 62–67% of consecutive tokens selected the same expert at middle layers. This locality creates caching optimization opportunities, but in an expert parallelism setup it also concentrates load on specific GPUs.
Mixture of Tokens: The Layer Chooses Which Tokens to Process
The idea is that not every layer needs to process every token. Each layer selects which tokens it will handle at that step, skipping the rest.
In Antoniak et al.'s implementation, this selection is continuous. The layer constructs its input by forming a weighted mixture of tokens from the batch, where the weights themselves are learnable parameters. Unlike sparse MoE, this is fully differentiable — which matters structurally. Conventional discrete top-k routing has points where gradients are cut off, requiring workarounds via auxiliary losses. MoT has no such constraint.
On the performance side, the paper reports 3× faster training than a dense Transformer and accuracy comparable to a similarly-sized sparse MoE. It also introduces a technique called transition tuning, which provides a path for converting trained MoE weights into a MoT architecture. This approach opens the door to converting existing MoE models to MoT without training from scratch, though how viable this path is in practice within the serving ecosystem has not been well validated yet.
The serving challenges MoT creates are different in kind from MoE's. Without expert parallelism, the all-to-all inter-GPU communication disappears. Instead, the variable number of tokens processed per layer destabilizes the KV cache structure. If a given layer processes only 60% of a sequence's tokens, that layer's KV cache does not align with the full sequence length. When each sequence in a batch processes a different number of tokens per layer, padding becomes irregular. This is why existing serving stacks built on static KV cache allocation — PagedAttention included — do not natively support MoT.
Serving Comparison: Memory, Batching, and Latency
| MoE | Mixture of Tokens | |
|---|---|---|
| Full parameter memory | Must reside on GPU entirely | Same (no savings) |
| Inter-GPU communication | All-to-all (every layer) | None |
| Batching bottleneck | Token distribution imbalance | Variable token count per layer |
| KV cache shape | Uniform across layers | Varies per layer |
| Mainstream framework support | vLLM, TensorRT-LLM, etc. | Not yet supported |
MoE is a relatively better fit for throughput-heavy workloads. With large batches and short sequences, tokens are more likely to spread evenly across experts, and all-to-all communication overhead shrinks relative to compute cost. In online inference scenarios with small batches and only a handful of tokens arriving at a time, communication cost can overtake compute cost.
MoT is theoretically better suited for small-scale deployments or edge inference, since it works on a single GPU and carries no inter-GPU communication overhead. In practice, however, the absence of a serving stack that can handle variable-token structures makes direct empirical comparison difficult. Looking only at active parameter counts, both architectures appear to win over dense — but what actually determines serving decisions in MoE is "how many GPUs can the full parameter set be distributed across," while in MoT it is "how efficiently can the batch scheduler handle variable-token structures."
Using Both Together
Applying MoE and token-level sparsity simultaneously should, in theory, compound the FLOPs reduction. In practice, the scheduler has to handle two routing decisions sequentially. The result of the first stage (token selection) must be available before the second stage (expert selection) can begin — a sequential dependency that stalls the GPU pipeline and layers routing overhead on top of scheduling complexity. This makes it hard to predict gains without empirical measurement, and in serving systems that combine both techniques, the benefits do not add up linearly.