Mixture of Experts vs Mixture of Tokens: Key Differences and Use Cases

Dense transformers activate every parameter for every token. A 175B model runs the full 175B network for each token it processes, and compute cost scales linearly with model size. Sparse activation came from a desire to decouple parameter count from FLOPs: keep many parameters, but use only a subset of them at inference time.

Both MoE (Mixture of Experts) and MoT (Mixture of Tokens) implement this philosophy, but they attack the problem from different angles. MoE asks "which expert should this token go to?", while MoT asks "which tokens should be mixed together for this expert?". The subject and direction of routing are inverted.

Mixture of Experts

The MoE architecture is straightforward. The FFN inside each Transformer layer is replaced with N expert FFNs, and a gating network (router) selects K of them for each token. The remaining N−K experts do no computation for that token.

The router takes each token's hidden state, computes probabilities over all N experts, and picks the top K:

$$G(x) = \text{TopK}(\text{softmax}(W_g \cdot x))$$

The final layer output is the gating-weighted sum of the K selected experts' FFN outputs. Switch Transformer (Fedus et al., 2021) pushed this to the extreme with Top-1 routing and was the first paper to demonstrate large-scale success with this approach. It achieved 7× faster pre-training than T5 and opened the door to scaling parameter counts into the trillions.

Mixtral 8x7B is the most well-known MoE implementation today. It uses 8 experts per layer with Top-2 routing: total parameters are 46.7B, but only ~13B are active at inference time. The FLOPs per forward pass are comparable to a 14B dense model, yet it competes with Llama 70B in quality.

Expert Collapse and Auxiliary Loss

The biggest practical problem with Top-K routing is router imbalance. As training progresses, the router starts sending tokens disproportionately to certain experts, while the rest receive too few updates and degrade into dead parameters. This is expert collapse. In severe cases, only 1–2 experts out of N remain functionally active, and the entire MoE design is effectively wasted.

The standard remedy is an auxiliary load-balancing loss. The Switch Transformer formulation is:

$$\mathcal{L}{aux} = \alpha \cdot N \cdot \sum{i=1}^{N} f_i \cdot P_i$$

$f_i$ is the fraction of tokens in the current batch actually routed to expert $i$; $P_i$ is the mean probability the router assigns to expert $i$. When both are large simultaneously, the loss increases, pressuring the router to spread load evenly. The recommended value is $\alpha = 0.01$.

A PyTorch implementation:

import torch
import torch.nn.functional as F

def router_auxiliary_loss(router_logits, n_experts, alpha=0.01):
    """
    Switch Transformer 방식 로드 밸런싱 보조 손실.
    router_logits: [T, N]  (T = 배치의 총 토큰 수, N = 전문가 수)
    """
    probs = F.softmax(router_logits, dim=-1)  # [T, N]

    # P_i: 라우터가 각 전문가에 할당한 평균 확률
    router_prob = probs.mean(dim=0)           # [N]

    # f_i: 실제로 각 전문가에 라우팅된 토큰 비율 (top-1 기준)
    top1_idx = probs.argmax(dim=-1)           # [T]
    token_fraction = F.one_hot(top1_idx, n_experts).float().mean(dim=0)  # [N]

    return alpha * n_experts * (token_fraction * router_prob).sum()

ST-MoE (Zoph et al., 2022) systematized these stabilization techniques and was the first to achieve state-of-the-art fine-tuning performance with a 269B sparse model.

In distributed training, communication overhead is an additional concern. When experts are sharded across GPUs (expert parallelism), each token must be routed to the GPU holding its assigned expert, and at large batch sizes the all-to-all communication cost becomes non-trivial.

Mixture of Tokens — Inverting the Routing Direction

MoT keeps the same expert structure but inverts the direction of routing. Instead of tokens selecting experts, each expert receives a mixed input drawn from tokens across multiple sequences in the batch.

The mechanism from the MoT paper (NeurIPS 2024) works as follows. A controller network computes softmax-based importance weights for each token. Tokens from multiple examples at the same sequence position are blended with expert-specific weights into a single input vector, which is then passed through the expert FFN. The output is redistributed back to the original tokens using the same weights.

The fundamental difference is that routing decisions are soft (continuous). MoE's Top-K selection is non-differentiable, so gradients don't flow cleanly through the router. MoT uses a weighted average, so end-to-end differentiation works without any special treatment. Load balancing is automatic by construction, and expert collapse cannot occur structurally — every expert has access to the full token pool in the batch.

There is also no token dropping. MoE drops or pads tokens that exceed the expert buffer capacity; MoT's weighted-sum structure means every token contributes to every expert's input.

The numbers reported in the paper: matching vanilla Transformer quality at 3× faster wall-clock training time and ¼ the FLOPs. The number of training steps to reach the baseline model's final loss is just 24% of the original.

Autoregressive decoding does come with constraints. Because MoT mixes tokens from different sequences, causal masking requires restricting mixing to tokens at the same sequence position. The benefit is maximized when the batch is large enough to provide a wide mixing pool; with a single sample at inference, there are no other tokens to mix with and the advantage largely disappears.

MoE vs MoT: Which Bottleneck Does Each Target?

Comparing the two approaches directly:

What MoE actually delivers is domain specialization. Different experts handling different input patterns allows the model to encode more diverse knowledge than a dense model with the same parameter count — Mixtral's numbers (46.7B total, 13B active) are a direct result. The cost is engineering overhead to maintain that specialization: auxiliary loss coefficient tuning, expert capacity configuration, and all-to-all communication in distributed setups all require attention.

What MoT delivers is training efficiency. It focuses on stability and convergence speed, and eliminates routing imbalance as an engineering problem at the architectural level. However, whether it achieves the same kind of explicit domain specialization as MoE, and how its batch-dependency affects serving, remain relatively underexplored.

Mixture-of-Depths (Raposo et al., 2024) sits on the same spectrum. Rather than experts, it skips entire layers on a per-token basis — tokens decide "which layers to compute". MoE determines which expert, MoT determines which tokens, and MoD determines which layers to process sparsely, each operating along a different dimension.

When to Use Which

As the HuggingFace MoE guide confirms, MoE is a proven production choice. Major open-source models — Mixtral, Qwen, DeepSeek — are MoE-based, and serving library support (including vLLM) is mature. If your workloads have high domain diversity and you have infrastructure to distribute across multiple GPUs, MoE is a viable choice right now.

MoT becomes relevant when training cost is the overriding concern — in the experimental phase, or when you can't afford to spend time on expert collapse and auxiliary loss tuning. For batch-inference API servers, MoT's batch dependency is actually an advantage. For single-request latency-sensitive on-premise single-GPU deployments, MoE's predictable inference cost is the better fit.

If you're interested in MoT, the realistic recommendation right now is to evaluate it in experimental settings only. The point at which serving library support reaches parity with MoE will be the real inflection point for broader adoption.