Transformer의 연산 비용을 줄이는 방법 중 가장 성공한 두 아이디어가 MoE(Mixture of Experts)와 sparse attention이다. 요즘 "Mixture of Tokens"(MoT)라는 표현도 종종 보이는데, 이건 별도의 논문 이름이 아니라 sparse attention 계열의 접근을 MoE와 대비하기 위해 쓰는 개념적 호칭에 가깝다. 공통점은 희소성이고, 차이는 희소성이 적용되는 레이어다. MoE는 FFN에, MoT류는 Attention에. 이 차이가 학습 비용에서는 어느 정도 추상화되지만, 추론 서빙 설계에 들어가면 완전히 다른 문제를 만든다.
The two most successful ideas for reducing Transformer compute costs are MoE (Mixture of Experts) and sparse attention. The term "Mixture of Tokens" (MoT) has been appearing more often lately — it isn't a standalone paper title, but rather a conceptual label used to contrast sparse-attention-style approaches against MoE. Both share sparsity as a common thread; they differ in which layer that sparsity is applied to. MoE targets the FFN; MoT-style approaches target Attention. This distinction is somewhat abstracted away at training time, but when you get into inference serving design, it surfaces as two fundamentally different sets of problems.
MoE: The Cost Structure of FFN Sparsification
Take Mixtral 8x7B as the reference point. A top-k router selects 2 of the 8 FFN experts per layer, and the selected experts' outputs are combined via a softmax-weighted sum. Total parameters are 46.7B, but only 12.9B are activated in a single forward pass. In terms of FLOPs, you're running a 47B-capacity model at roughly the compute cost of a 13B model.
At inference time, all experts must reside in GPU memory — you can't know which expert a token will be routed to before the forward pass runs. Loading in fp16 requires roughly 93 GB, which doesn't fit on a single A100 80 GB card. This structural property — using FLOPs selectively while occupying full memory — is the starting point for any MoE serving design.
Mixture of Tokens: Sparsifying the Attention Layer
There's no single paper called "Mixture of Tokens." Just as FFN sparsification gets called MoE, MoT is an umbrella term for architectures that apply MoE-style token selection to the attention computation. Concrete implementations appear under several different names.
The most direct implementation is MoBA (Mixture of Block Attention), published by Moonshot AI in 2025. It partitions the context into fixed-size blocks, and each query token attends only to the top-K most relevant blocks selected via a gating mechanism. Where dense attention is O(n²), MoBA is O(n·K·b) — n being sequence length, K the number of selected blocks, and b the block size. At the time of publication, it was already deployed in production handling Kimi's long-context requests.
DeepSeek's Native Sparse Attention (NSA) consists of three parallel branches: a coarse-grained branch that compresses contiguous blocks into a single representation, a selection branch that processes high-importance blocks at full resolution, and a local branch that attends to a sliding window of recent tokens. Their outputs are summed to produce the attention output. The "hardware-aligned" in the paper title is there for a reason — without hardware-aligned kernels, sparse attention can be slower than dense attention in practice, despite the theoretical FLOPs reduction.
Earlier sparse attention work like Longformer and BigBird addresses the same problem, but uses fixed patterns such as sliding windows plus global tokens. MoBA and NSA use a router that inspects the input content and dynamically decides which tokens to attend to. This dynamic routing is what makes MoT-style architectures work — and simultaneously what introduces new problems in serving.
FFN Sparsity vs. Attention Sparsity: Serving Implications
The serving challenges these two structures create are fundamentally different in character.
| Dimension | MoE (FFN sparsification) | MoT (Attention sparsification) |
|---|---|---|
| What is sparsified | FFN expert selection | Attention target tokens/blocks |
| Memory footprint | All parameters must reside in memory | KV cache size can be reduced |
| Prefill cost | FFN compute reduced (k/N) | Quadratic sequence-length term reduced |
| Routing pattern | Fixed expert set per layer | Different attention pattern per request |
| Batching efficiency | Requires grouping tokens by expert | Dynamic patterns make batch alignment hard |
| KV cache reuse | Standard prefix caching applies | Low predictability due to dynamic patterns |
The core batching challenge in MoE is token scatter. When tokens in the same batch route to different experts, all-to-all communication occurs between the GPUs holding those experts. vLLM addresses this with the --enable-expert-parallel flag for Expert Parallelism, and mitigates token imbalance across experts with EPLB (Expert Parallel Load Balancer).
vllm serve deepseek-ai/DeepSeek-V3-0324 \
--tensor-parallel-size 1 \
--data-parallel-size 8 \
--enable-expert-parallel
MoT-style architectures run into a different wall. When the set of attended tokens varies per request, it's hard to predict ahead of time which KV cache entries will be reused. Prefix caching operates on a simple rule — "cache hit if the first n tokens are identical" — but in a dynamic attention pattern environment, even identical prefixes may attend to different blocks in subsequent computation. Prefill scheduling faces the same issue: if requests in a batch have different attention block patterns, it becomes difficult to fuse their computation.
The upside shows up clearly in long-context settings. With a sequence length of 32K tokens, if each query attends to only the top 2048 tokens, attention FLOPs drop to 1/16 of the full cost. This saving is independent of MoE's FFN reduction. The two designs aren't competing — they're composable.
Combined Architectures and Current Serving Support
DeepSeek-V2 demonstrates this combination in practice. The FFN is sparsified with the DeepSeekMoE architecture, while attention uses MLA (Multi-head Latent Attention) to compress KV into low-dimensional latent vectors. This reduces the KV cache by 93.3% compared to standard MHA, while activating only 21B of the model's 236B total parameters per forward pass.
The gap in serving engine support is still wide. MoE has a full stack in vLLM: Expert Parallelism, WideEP, DeepEP backends, and EPLB. Serving Mixtral and DeepSeek-V3-scale models across multiple GPUs is already routine. Sparse attention is a different story. MoBA's code is public and running in Kimi's production environment, but neither vLLM nor SGLang provides generalized kernels that handle arbitrary sparse attention patterns efficiently. For theoretical FLOPs reductions to translate into real latency gains, sparse attention needs kernel-level optimization on par with FlashAttention. NSA's paper takes a step in that direction, but given how long it took MoE serving to reach its current state, there's still ground to cover.