Mixture of Experts: Cutting Compute Without Cutting Parameters

Mixture of Experts Deep Dive: Reducing Compute Without Reducing Parameters

Multiple studies have established that LLM performance scales with parameter count, which has driven Meta, Google, OpenAI, and others to race toward ever-larger models. But as models grow, so do the compute costs of training and serving them. Mixture of Experts — MoE — emerged as a way to train and serve massive LLMs at lower cost.

One of the most prominent MoE models is Mistral AI's Mixtral-8x7B. OpenAI's GPT is widely suspected to use a MoE architecture as well, though the details have never been confirmed. These MoE models run inference using only a fraction of their total parameters.

MoE: Basic Structure and How It Works

The core concepts behind MoE are:

• Expert networks (Experts): A MoE model consists of multiple independent neural networks, each called an "expert." In practice, a model may contain hundreds to thousands of experts.
• Gate network (Router): A network that decides which experts to activate for a given input. The router takes the input, selects a subset of experts, and combines their outputs — typically via a weighted sum — to produce the final output. Only a few experts are activated per input; the rest remain inactive.

MoE in the Transformer Architecture

A standard Transformer LLM follows the pattern: Self-Attention → Add & Norm → FFN → Add & Norm. MoE modifies the FFN stage: instead of a single FFN, there are multiple FFN experts. After a token passes through Self-Attention → Add & Norm, the router directs it to one or more specific FFN experts. The next token may be routed to a completely different set of experts.

The number of experts a token passes through is controlled by a top-k setting. When top-k > 1, the token passes through k experts independently, and their outputs are combined — typically via a weighted sum using the router's output probabilities — to produce the final result.

Routing Strategies

The main MoE routing variants are:

• Standard MoE: Combines the outputs of all experts. Computationally expensive, but makes full use of every expert's capacity.
• Switch Transformer: Routes each input to exactly one expert. Very efficient, but risks information loss.
• GShard: Google's approach for applying MoE at scale, designed for strong scalability across diverse tasks.
• BASE Layers: Trains with a balanced load across experts, ensuring all experts are utilized roughly equally.
• HashMoE: Uses a hash function to assign experts. Computationally efficient, but less flexible.
• Soft MoE: Softly blends the outputs of all experts — a middle ground between Standard MoE and Switch Transformer.
• ST-MoE (Sparse Top-k MoE): Selectively activates the top-k experts out of many for each input, balancing efficiency and performance in large-scale language models.

Other approaches include Expert Choice Routing and Mixture of Experts with Expert Choice (MoEC).

Token Choice vs. Expert Choice

MoE routing broadly falls into two paradigms.

Token Choice: The standard approach, where each token independently selects its top-k experts. This integrates naturally with autoregressive generation, but popular experts tend to attract disproportionate traffic, causing load imbalance. Common mitigations include overprovisioning capacity by a factor of 2–8× and adding an auxiliary balancing loss.

Expert Choice: An approach proposed by Google in 2022 that inverts the routing direction — each expert selects the top-k tokens from the batch rather than the other way around. This mathematically guarantees perfect load balancing and allows variable expert assignment per token. The downside is that expert selection requires access to the full sequence, which violates causality and makes it incompatible with standard autoregressive language modeling as-is.

Major MoE Models: 2024–2025 Comparison

DeepSeek-V3 introduced a loss-free balancing approach that replaces the auxiliary loss with a per-expert dynamic bias term added to the routing scores. Overloaded experts get their bias reduced; underutilized experts get it increased. Crucially, this bias is not carried into the final gating weights, so it has no effect on output quality. Qwen3-MoE takes a different path — no shared expert, relying solely on a global-batch load balancing loss to encourage expert specialization.

Code Example: Mixtral-8x7B

Here is how to run inference with Mixtral-8x7B, one of the most widely used MoE models:

from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistral_common.protocol.instruct.messages import UserMessage
from mistral_common.protocol.instruct.request import ChatCompletionRequest

# Tokenization

mistral_models_path = "MISTRAL_MODELS_PATH"
 
tokenizer = MistralTokenizer.v1()
 
completion_request = ChatCompletionRequest(messages=[UserMessage(content="Explain Machine Learning to me in a nutshell.")])
 
tokens = tokenizer.encode_chat_completion(completion_request).tokens

# Inference by mistral_inference

from mistral_inference.transformer import Transformer
from mistral_inference.generate import generate

model = Transformer.from_folder(mistral_models_path)
out_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)

result = tokenizer.decode(out_tokens[0])

print(result)

# Inference with huggingface transformers

from transformers import AutoModelForCausalLM
 
model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x7B-Instruct-v0.1")
model.to("cuda")
 
generated_ids = model.generate(tokens, max_new_tokens=1000, do_sample=True)

# decode with mistral tokenizer
result = tokenizer.decode(generated_ids[0].tolist())
print(result)

Mixture of Tokens (MoT)

Where MoE asks "which expert should handle this token?", Mixture of Tokens (MoT) approaches the problem from the opposite direction. Presented at NeurIPS 2024, MoT doesn't route tokens to experts — instead, it first mixes tokens from multiple examples in the batch using learned weights, then feeds those mixed representations into the expert FFNs.

The three key ideas are:

• At each layer, tokens from different examples in the batch are combined into a mixed representation via a learned weighted sum.
• This mixed representation is what passes through the expert FFN.
• The mixing weights are learned; ablations in the paper show this outperforms a fixed uniform weighting (1/n).

The main differences from MoE:

• MoT is fully continuous — there are no discrete routing decisions, so there is no gradient discontinuity problem.
• When the number of experts equals the number of tokens, total compute is the same as top-1 Token Choice MoE.
• Experiments in the paper show improvements in training stability and expert utilization compared to both Token Choice and Expert Choice sparse MoE.

Because MoT mixes tokens across different examples in the batch, it operates as designed in batched inference settings rather than single-request scenarios.

Fine-Tuning MoE Models: What to Watch Out For

Expert Collapse

Fine-tuning a pretrained MoE model often triggers routing collapse, where the router funnels most tokens to a small number of experts. The remaining experts receive almost no gradient signal, effectively shrinking the model's usable capacity.

Load Balancing Auxiliary Loss

The standard mitigation is to add an auxiliary loss that penalizes high variance in the number of tokens each expert receives. This is added on top of the primary task loss. However, if the auxiliary loss coefficient is too large, it creates gradient interference that conflicts with the task learning objective and can hurt performance. For fine-tuning, a coefficient in the range of 0.001–0.01 relative to the primary loss is typical.

Loss-Free Balancing

DeepSeek's 2024 loss-free approach maintains expert balance without an auxiliary loss by adding a per-expert dynamic bias term to routing scores. Overloaded experts have their bias reduced; underutilized experts have it increased. This bias does not feed into the final gating weights. It's worth considering for fine-tuning scenarios where you want expert balance without the gradient interference that comes with auxiliary losses.

Serving MoE in Production: vLLM

The parallelization strategy you choose when serving MoE models with vLLM has a significant impact on both memory efficiency and throughput.

Tensor Parallelism vs. Expert Parallelism

• Tensor Parallelism (TP): Configured via --tensor-parallel-size. Shards the weight tensors of all layers — including attention — across all GPUs.
• Expert Parallelism (EP): Enabled with the --enable-expert-parallel flag. Distributes experts across GPUs at the expert level, but only for MoE layers. Uses the same degree of parallelism as the configured tensor_parallel_size.

# Mixtral 8x7B — 4-GPU Expert Parallelism example
vllm serve mistralai/Mixtral-8x7B-Instruct-v0.1 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel

MLA-Based Models (DeepSeek Family)

Applying standard TP to models that use Multi-head Latent Attention (MLA), like DeepSeek-V3, causes the KV cache to be replicated across every GPU, wasting significant memory. The vLLM documentation recommends a hybrid approach for these models: Data Parallelism for the attention layers and Expert Parallelism for the expert layers.

# DeepSeek-V3 — 8-GPU DP+EP hybrid example
vllm serve deepseek-ai/DeepSeek-V3 \
  --tensor-parallel-size 1 \
  --data-parallel-size 8 \
  --enable-expert-parallel