Sparse activation has a clear motivation. Applying every parameter to every token at every Transformer layer is wasteful. Given a fixed compute budget, it's better to spread that budget across more parameters or concentrate it where it's actually needed.
MoE (Mixture of Experts) and MoT (Mixture of Tokens) share this motivation. They differ only in the axis along which sparsity is applied. MoE partitions the parameter space — activating only a subset of experts per token. MoT constructs the input each expert processes as a weighted mixture of tokens drawn from multiple examples in the batch. This single difference completely changes the character of memory and routing costs at serving time.
MoE: Structure and Real Inference Costs
MoE's structure is straightforward. The Transformer FFN layer is replicated into N experts, and a router selects k of them per token. Fedus et al. (2022) Switch Transformers was the first large-scale experiment to push this to k=1, reducing routing cost to the extreme. Mixtral-8x7B uses a k=2 configuration, selecting 2 of 8 experts.
The router takes a token embedding, computes softmax scores, and sends the token only to the top-k experts:
gate_scores = softmax(W_g @ x) # shape: [num_experts]
selected = top_k(gate_scores, k=2)
output = sum(gate_scores[i] * FFN_i(x) for i in selected)
In this setup, per-token FLOPs drop to k/N of a dense model. For Mixtral, that's 2/8 = 25%. With k=1, FLOPs halve, but load imbalance — tokens concentrating on specific experts — becomes more severe. k=2 distributes load better and produces more stable quality, but at twice the routing and communication cost.
The real burden is memory. Mixtral-8x7B has 46.7B total parameters, with roughly 12.9B active per token. But since it's impossible to know in advance which expert any token will select, all 8 expert weight matrices must reside in GPU memory at all times. In FP16, that's roughly 87 GB. A single A100 80 GB card can't serve this model. The 12.9B active-parameter figure has no bearing on VRAM requirements whatsoever.
Addressing this requires distributing across multiple GPUs via tensor parallelism (TP) or expert parallelism (EP). EP places each expert on a separate GPU and routes tokens to whichever GPU holds their selected expert. This introduces all-to-all communication — tokens in a batch scatter to their respective expert GPUs, compute, then reassemble. Over NVLink this cost is manageable; over PCIe it becomes a throughput bottleneck. vLLM officially supports EP, integrates DeepEP kernels to reduce all-to-all overhead, and exposes the --enable-eplb flag for expert load balancing.
Load imbalance is a separate problem. When the router funnels too many tokens to a single expert, tokens that exceed capacity are skipped. Switch Transformers sets the capacity factor to 1.25 by default:
capacity = ceil(1.25 × tokens_in_batch / num_experts)
Tokens beyond this limit are passed through via the residual connection without being processed. Increasing the capacity factor reduces overflow but wastes memory; decreasing it causes token dropping and quality degradation. An auxiliary load balancing loss is commonly used to encourage the router to distribute tokens evenly, but the tradeoff remains — this auxiliary objective competes with the primary training goal.
None of this means the FLOP reduction is meaningless in practice. Mixtral-8x7B performs roughly 5× fewer operations than Llama 2 70B on a per-active-parameter basis, which translates to lower per-request latency. But the cost of realizing that advantage — in memory and distributed communication — is the central challenge from a serving perspective.
Mixture of Tokens: How Tokens Get Mixed
Antoniak et al. (2023) MoT takes a fundamentally different approach to sparsity. Where MoE replicates parameters N times and discretely routes tokens to individual experts, MoT constructs each expert's input as a weighted mixture of tokens drawn from different examples within the batch.
In MoT, each expert doesn't receive tokens from a single sequence — it processes a representation that blends tokens from multiple sequences in the batch, with mixing weights learned by the router. This is why it's sometimes called "Continuous MoE": rather than discretely assigning tokens to experts, it continuously determines mixture ratios at the batch level.
Parameter efficiency is high. The paper reports a 3× training speedup over a dense Transformer, with performance comparable to MoE architectures of the time. Because expert weights aren't replicated N times, parameter count and GPU memory requirements don't grow.
The tradeoff is cross-example dependency within the batch.
In autoregressive generation, if the current token from request A is mixed with tokens from request B before processing, then request A's output depends on the batch composition. The same prompt can produce slightly different results depending on what other requests happen to be batched alongside it. Inference that should be deterministic becomes a function of batch construction.
The KV cache problem is more concrete. vLLM's PagedAttention manages each request's KV cache as an independent unit. If token representations in MoT are influenced by other tokens in the batch, this per-request independence assumption breaks down. Caching prefill results and processing tokens independently during decode no longer hold. Continuous batching — whose flexibility depends on being able to add or remove requests from a batch at any time — is difficult to maintain when tokens within a batch are interdependent.
Direct Tradeoff Comparison
| MoE | MoT | |
|---|---|---|
| Weight memory | expert count × FFN size (all must reside in memory) | Same as a dense model |
| Routing | Discrete, top-k expert selection | Continuous, weighted token mixing across batch |
| Per-request independence | Fully independent | Depends on batch composition |
| KV cache management | Per-request, PagedAttention compatible | Complex due to batch-level dependencies |
| Distributed communication | EP required, all-to-all | Single GPU sufficient |
| Load balancing | Capacity factor + auxiliary loss required | Not needed with continuous mixing |
| Production stack support | vLLM, TRT-LLM officially supported | No general-purpose support |
For small batches or single-GPU environments, MoT has the advantage. No N× parameter replication means no memory overhead, and EP with all-to-all communication is unnecessary. For production serving under high concurrent load, MoE is easier to operate. Requests are fully independent, so continuous batching works naturally and KV caches can be managed per-request. EP is complex and memory-hungry, but vLLM and TRT-LLM already absorb that complexity.
The Reality in Today's Serving Stacks
vLLM officially supports Expert Parallelism, handling the main complexities of MoE serving through DeepEP kernel integration and EPLB (Expert Parallel Load Balancer). TensorRT-LLM includes MoE-specific plugins as well. Models like Mixtral and DeepSeek-V2 are already served in production through this path.
MoT is a different story. Cross-example token mixing doesn't align with vLLM's PagedAttention or continuous batching architecture. You can't simply drop it into the same inference stack — batch management and KV cache logic would need to be redesigned from scratch. As of now, there is no general-purpose inference framework that supports MoT.