Both architectures start from the same premise: not every parameter needs to be active for every computation. But which axis sparsity is applied along determines everything about the inference system. MoE activates only a subset of experts in the FFN layers; MoT prunes the token-to-token interactions in attention itself. That single difference drives memory requirements, batching efficiency, and load-balancing problems in entirely different directions.
Why Sparse Models
In a dense model, every parameter is engaged at inference time. More parameters means proportionally more memory and FLOPs, so improving model quality drives cost up super-linearly. Sparse models break that coupling — total parameter count grows, but the number of parameters actually activated per token is bounded. The tradeoff is that the choice of which parameters to skip determines what kind of problems the inference system has to deal with.
Mixture of Experts: Architecture and Real Inference Costs
In MoE, the FFN in each Transformer layer is replaced by a set of experts — independent MLP networks. Each time a token passes through a layer, a router (gate) selects the top-k experts and produces output as a weighted sum of their results.
Mixtral 8x7B is currently the most widely referenced open MoE model. It selects top-2 out of 8 experts, giving a total of 46.7B parameters but only ~12.9B active parameters per token. This looks efficient on paper, but naively interpreting these numbers in a serving context leads to trouble.
Memory is allocated for all parameters. Since there is no way to know ahead of time which experts will be activated in which layers during inference, every expert's weights must reside in memory. The actual VRAM requirement for Mixtral 8x7B in bf16 is roughly 87 GB — comparable to a 46.7B dense model, not the ~26 GB you'd need for a dense model with 12.9B active parameters. Taken to an extreme, this gives you something like DeepSeek V3: 671B total parameters, 37B active — compute at the 37B level, but weights alone consume 1.3 TB in BF16.
Expert Parallelism and Communication Overhead
When fitting all experts onto a single GPU or node becomes infeasible, expert parallelism (EP) is used. Each GPU owns a different subset of experts, and tokens are dispatched to the appropriate GPU based on routing decisions. The problem is that this requires all-to-all communication. Unlike the all-reduce used in tensor parallelism for dense models, all-to-all has a communication pattern that changes with every forward pass, driven by per-token routing results. Inter-node InfiniBand bandwidth must be able to sustain this traffic; at large scale, there are reports of needing aggregate throughput on the order of 2 TB/s.
Load Imbalance: The Core Pain Point of MoE Serving
The main reason MoE is difficult to serve is load imbalance. The router is trained with an auxiliary loss to encourage uniform expert utilization, but real inference traffic causes tokens to cluster on certain experts. Since Switch Transformer, MoE research has combined capacity factors with auxiliary losses to mitigate this, but has not eliminated it. Recent work observes that even well-trained MoE models exhibit significantly imbalanced routing. In an EP setup, when tokens pile onto a particular GPU, that GPU becomes the bottleneck while the rest wait. Applying a dynamic redistribution algorithm (LLEP) has shown throughput improvements of up to nearly 5×, which is itself evidence of how severe the imbalance can be.
Mixture of Tokens: What Changes When Sparsity Moves to the Token Axis
MoT-style approaches move sparsity from the FFN to attention. Instead of choosing which expert to use, the model chooses which tokens exchange information with which other tokens. Standard self-attention computes relationships for every pair of tokens in the sequence — O(n²) cost — so selectively restricting these token-to-token interactions reduces both prefill cost and KV cache size.
Several implementations exist. Some operate block-wise, partitioning the attention span — the MOBA (Mixture of Block Attention) family. Others learn different token subsets per head — Mixture of Sparse Attention (MoSA). Token Sparse Attention processes Q, K, V over a compressed token set and reconstructs the attention output, reporting a 3.23× speedup on attention computation at 128K context with less than 1% accuracy degradation.
Parameter memory stays the same as the equivalent dense model. What shrinks is the cost proportional to sequence length — attention compute during prefill and KV cache size during decoding. A smaller KV cache means more sequences fit in the same VRAM, which increases effective batch size. This distinction matters in long-context serving.
The catch is that when token selection patterns differ across sequences, batching becomes more complicated. Dense attention applies the same causal mask to every sequence in a batch, but sparse attention produces a different masking pattern per sequence or even per head. From the GPU's perspective, instead of running a single monolithic GEMM, it must handle multiple sub-operations with varying patterns — which can actually be slower than dense attention in a naive implementation. Translating theoretical sparsity into measured throughput gains requires low-level kernel optimization on par with FlashAttention.
Serving Comparison
A direct comparison of the serving burden between the two architectures:
| Dimension | MoE | MoT |
|---|---|---|
| Parameter memory | All expert weights (several× a dense model) | Same as dense |
| Active FLOPs | Reduced to active experts | Reduced proportional to sparsity ratio |
| Communication pattern | All-to-all (in EP setups) | All-reduce (same as dense) |
| Load balancing | Expert skew causes GPU idle time | Local to head/layer |
| KV cache | No structural change | Can be reduced by sparsity ratio |
| Prefill cost | FFN savings, attention unchanged | Attention savings, FFN unchanged |
| Framework support | Broad (vLLM, TGI, SGLang) | Limited (requires custom kernels) |
MoE's burden concentrates at deployment scale: the number of nodes required to host the model, the inter-node communication infrastructure, and the overprovisioning needed to absorb load imbalance. MoT's burden is in kernel engineering. Making sparsity deliver its theoretical gains requires low-level optimization to handle the GPU's irregular access patterns.
The available model ecosystem strongly favors MoE today. There are multiple production-validated MoE models — Mixtral, Mixtral 8x22B, DeepSeek V3, Llama 4 — and both vLLM and SGLang support MoE serving with EP. Most MoT implementations are still research-stage, and few models in this family have native support from serving frameworks.
Because the two architectures apply sparsity along orthogonal axes, they can in principle be combined — activating only a sparse subset of experts in the FFN while also restricting token interactions in attention. In practice, however, managing two independent dynamic routing complexities simultaneously carries significant systems cost, and no regime has yet established that the combined benefit justifies it.