There are two broad approaches to scaling up transformers while keeping inference costs down. One is to maintain multiple experts and activate only a subset of them; the other is to reduce the number of tokens processed at each layer. The former is Mixture of Experts (MoE); the latter's canonical implementation is Mixture-of-Depths (MoD). Both are instances of the same "sparse activation" idea, but what gets skipped determines an entirely different set of problems for LLM serving systems.
Two Different Axes of Sparsity
MoE takes sparsity in parameter space. At each layer, every token selects the top-k experts from the full set and routes through them. All tokens pass through all layers, but only a subset of experts is activated within each FFN block.
Mixture-of-Tokens (MoT), or the MoD family, takes sparsity in sequence space. The layer selects which tokens to process. All parameters are used, but a significant fraction of the sequence skips the self-attention and MLP of that layer entirely. This approach is also called conditional computation.
| MoE | MoT/MoD | |
|---|---|---|
| Sparsity axis | Parameters (experts) | Sequence (tokens) |
| Selection direction | Token → Expert | Layer → Token |
| Tokens processed | All, at every layer | Subset, per layer |
| VRAM for weights | Full model must reside in memory | Same as dense |
The Cost Structure of MoE Inference
Mixtral 8x7B has 46.7B total parameters and 12.9B active parameters per token. Because each layer uses top-2 expert routing out of 8, the per-inference FLOPs are comparable to a 12.9B dense model. However, all 46.7B weights must remain in VRAM at all times, requiring roughly 93 GB in FP16 — more than a single A100 80GB can hold.
Beyond memory, the problem that comes up most often in real serving is expert load imbalance. When tokens in a batch concentrate on specific experts, those experts exceed their capacity and the overflow tokens are dropped.
expert_capacity = (tokens_per_batch / num_experts) × capacity_factor
A capacity_factor of 1.25 is a common starting point — reserving 25% headroom above the theoretical maximum — but drops still occur when requests skew heavily toward certain experts. ST-MoE (Zoph et al., 2022) mitigated the imbalance with an auxiliary load-balancing loss, but it does not eliminate skew entirely.
Batch size also has a large impact on throughput. In MoE, small batches mean few tokens per expert, which yields small GEMM matrices and low GPU utilization. MoE-Inference-Bench (2025) results showed roughly a 100× throughput increase when scaling batch size from 1 to 128. The smaller the batch, the more the sparsity of MoE works against you.
Distributed inference adds all-to-all communication overhead, since tokens must be sent to the GPU holding the responsible expert. vLLM mitigates this with a fused MoE kernel that combines expert selection, routing, and FFN computation into a single kernel, but benchmarks consistently show lower GPU utilization for expert parallelism compared to tensor parallelism.
Mixture of Tokens: Skipping Tokens Instead
The foundational work in this family is Raposo et al. (2024)'s Mixture-of-Depths. At each transformer layer, the number of tokens to process is fixed at k, and a learned router selects the top-k tokens to pass through self-attention and MLP. The remaining tokens bypass the layer via the residual connection only.
Because k is fixed ahead of time, the computation graph remains static — a practical implementation advantage, since XLA and CUDA graph optimizations apply without modification. The paper reports that configuring the model to process only 50% of tokens per layer achieves up to 1.5% lower training loss than a vanilla transformer at the same FLOPs budget. You halve the FLOPs per forward pass and get better performance.
Observing which tokens get skipped reveals a pattern. Whitespace, punctuation, and tokens that are highly predictable from surrounding context tend to skip shallower layers. Rare words, sentence boundaries, and semantically significant positions tend to be processed at more layers. Because the model effectively decides how much "thinking" each token warrants, this family is sometimes called MoT LLMs.
What Changes from a Serving Perspective
| Item | MoE | MoT/MoD |
|---|---|---|
| VRAM footprint | Large (full weights resident) | Same as dense |
| Prefill predictability | Varies with batch composition | Varies per token (less predictable) |
| Distributed communication overhead | Adds all-to-all | None |
| Batch processing efficiency | Better with larger batches | Relatively insensitive |
| Production serving maturity | High (vLLM, TRT-LLM, SGLang) | Research stage |
MoE demands more VRAM, but with large batches the load spreads across experts and GPU utilization improves. Mixtral and Grok-family models currently occupy this position in production deployments.
The challenge with MoT/MoD is prefill predictability. In a continuous batching environment, two requests with the same sequence length can have very different actual FLOPs depending on which tokens pass through which layers, making it hard to estimate prefill latency. This conflicts with the "same length = same cost" assumption baked into existing LLM serving architectures. KV cache allocation could shrink if only attended tokens are stored, but how paged attention handles non-contiguous token indices is implementation-dependent.
MoE serving shines in API server deployments with large batches and a roughly uniform request distribution. MoT/MoD is more attractive for single-GPU setups, latency-critical low-batch workloads, and long-context inference — longer sequences have a higher proportion of predictable tokens, so the skip efficiency improves. That said, as of 2025, no production serving framework exists that can deploy MoD in practice. Schedulers capable of handling irregular prefill and memory managers that deal with non-uniform KV caches have not yet been implemented.