Dense transformers treat every token identically. With 512 tokens, all 512 pass through the attention and FFN of every layer. As parameter count scales up, this uniform processing becomes a direct computational bottleneck.
The intuition behind sparse activation is straightforward. If you can decouple total parameters from the parameters actually activated during inference, you can preserve model capacity while reducing inference cost. The tradeoff is routing overhead and load imbalance.
There are two distinct answers to how you implement this idea. MoE and MoT differ in the axis along which they introduce sparsity, and that difference produces entirely different memory and compute cost profiles at inference time.
Mixture of Experts: Sparsity Along the Layer Axis
MoE splits each FFN layer into multiple "experts" and routes each token through only a subset of them. The number of layers stays the same, but the FFN within each layer is replaced by N parallel experts.
Routing operates independently per token. A gating network scores which experts a given token should be sent to, and only the top-k experts are activated. Mixtral-8x7B selects 2 experts out of 8 per layer (top-2). Total parameters are 46.7B, but active parameters at inference are 12.9B — the remaining 6 experts are completely inactive when processing that token.
Push the expert count to the extreme and the ratio widens further. DeepSeek-V2 uses 160 routed experts plus 2 shared experts per layer, activating only 6 per token. Of its 236B total parameters, only 21B are used to process each token — an active/total ratio of roughly 9%.
In this architecture, load balancing is a central training stability concern. If routing collapses onto a subset of experts (expert collapse), the rest converge in an undertrained state. Switch Transformer (Fedus et al., 2022) introduced top-1 routing and mitigated this with an auxiliary loss that regularizes routing weights to equalize the fraction of tokens assigned to each expert. The choice between k=1 and k=2 is not a minor detail — k=1 has higher variance and lower routing cost, while k=2 improves expert utilization at the expense of increased all-to-all communication overhead. This is precisely why Mixtral opted for k=2.
Mixture of Tokens: Sparsity Along the Token Axis
MoT works in the opposite direction. The layers are fixed — their number and structure remain unchanged — and instead, the model selects which tokens receive the heavy computation (full attention or FFN) at each layer. Tokens that are not selected skip that layer entirely via the residual connection.
Not all tokens within a sequence are equally important. Core nouns and verbs carry rich contextual information, while function words and punctuation follow comparatively simple patterns. Applying the same compute to all of them is the inefficiency MoT sets out to address.
Mixture of Depths (Raposo et al., 2024) is the canonical implementation of this approach. It fixes the number of tokens processed per layer at k, and a lightweight router passes only the top-k tokens through attention and MLP. Because the compute graph remains static, compatibility with existing CUDA kernels is straightforward, and the paper reports inference sampling speeds more than 50% faster than the baseline.
There is one critical difference from MoE. MoT adds no parameters. Because no new experts are created, total parameter count does not increase — instead, existing parameters are used more selectively. As a method for reducing compute without expanding model capacity, MoT is much closer to "pure" sparsification than MoE.
Putting Both Strategies Side by Side
| MoE | MoT | |
|---|---|---|
| Sparsity axis | Expert (FFN) selection within a layer | Token selection per layer |
| Routing question | Which experts to use | Does this token deserve compute at this layer |
| Layer structure change | FFN replaced by N parallel experts | Layers unchanged; tokens are filtered |
| Total parameter count | Increases with number of experts | Unchanged |
| Memory access pattern | Only selected expert parameters are active | Only selected tokens are computed |
| Batch imbalance type | Variance in tokens per expert | Variance in layers traversed per token |
| Primary serving bottleneck | Expert VRAM residency, distributed routing communication | Variable compute causing padding inefficiency in batches |
The total parameter count row matters. Because MoE scales parameters with the number of experts, Mixtral-8x7B's full size is 46.7B. At serving time, all 46.7B parameters must reside in VRAM — you cannot know in advance which expert the next token will select. The active parameter count of 12.9B is comparable to a 7B-class dense model in terms of compute, but memory requirements track the full model size.
What Changes in Inference Serving
The first problem with MoE serving is VRAM residency cost. All expert parameters must be loaded onto GPU, and a large expert count makes multi-GPU distribution unavoidable. In a distributed setting, each routing decision triggers all-to-all communication across GPUs. Larger batch sizes and more experts translate directly into TTFT and throughput degradation. This is why DeepSeek-V2 distributes its 160 routed experts across multiple GPUs.
MoT serving presents a different kind of problem. When tokens traverse different numbers of layers, the compute load per token varies within a batch. If token A passes through 10 layers and token B through 24, the shorter-path tokens must either wait or be padded to match. This padding inefficiency has a noticeable impact on TPOT (time per output token). On the other hand, there is no expert parameter loading problem — since total parameter count does not increase, VRAM pressure does not either.
From a TTFT perspective, MoE is more sensitive to expert routing communication latency, while MoT is more sensitive to batch reordering overhead. From a throughput perspective, MoE is bound by distributed communication bandwidth, while MoT faces higher kernel optimization complexity in handling variable per-token compute.
The Case for Combining MoE and MoT
The two approaches are not mutually exclusive. Applying sparsity along both the layer axis and the token axis simultaneously would, in theory, stack savings from both dimensions. Architectures that place a token-level router on top of an MoE layer, or that use MoE in some layers and MoD in others, are active areas of research.
A word of caution on practical serving complexity: optimizing schedulers and kernels that simultaneously handle expert parallelism communication and variable token paths to production-grade performance is still an open problem. At the intersection of both sparsity axes, the variables governing batch scheduling multiply in complexity.