DeepSeek R1: Architecture and Reinforcement Learning Deep Dive

DeepSeek is a long topic to cover.

1. DeepSeek LLM (https://arxiv.org/pdf/2401.02954)

2. DeepSeek MOE (https://arxiv.org/pdf/2401.06066)

3. DeepSeek-V2 (https://arxiv.org/pdf/2405.04434)

4. DeepSeek-V3 (https://github.com/deepseek-ai/DeepSeek-V3/blob/main/DeepSeek_V3.pdf)

5. DeepSeek-R1 (https://github.com/deepseek-ai/DeepSeek-R1/blob/main/DeepSeek_R1.pdf)

Each paper claims its own contributions, but since the methodology in R1 is what's really worth examining today, let's set the earlier papers aside for later.

Here's a rough overview.

DeepSeek is ultimately a Transformer-based architecture, so the core architectural changes amount to modifications of the Transformer Block.

Following the progression across the papers above, the changes accumulate gradually — which means if you jump straight to V3, you'll constantly run into phrases like "building directly on the architecture we established in V2 ~~".

The core of the V3 Transformer Block is already present in those two prior works.

First is the Shared Expert architecture.
The intuition is that certain patterns in the training data are likely to be used in a structurally uniform way across tasks.

Having shared experts that can absorb these common patterns reduces overall redundancy and allows parameters to be used more efficiently — or so the authors argue.

Next is MLA.
The core motivation here is an effective response to severe computation cost.

During inference, computing u_t — the output hidden state — requires the K and V values computed across all attention heads.

The authors note that this requires caching 2 * n_h * d_h * l elements per token, and argue this creates a significant bottleneck.

As the number of heads and the model dimension grow, the sheer volume of intermediate Q, K, V values that must be held in memory does indeed become a real issue.
**(Transformer computation is fundamentally structured around projecting the input hidden state through Q, K, V matrices to produce those intermediate values, after all.)

The authors address this with a single key idea:
if the bottleneck isn't the speed of the matrix operations themselves but rather the large amount of cache memory required, then the solution is to project those intermediate Q, K, V values into some latent space before storing them.

A similar idea appeared in Mamba, which I reviewed previously.

The HiPPO framework can also be viewed as a technique for mapping the output of a differential equation process into a latent space.

Anyway, the key idea is straightforward: use a down-projection matrix to compress Q, K, V values for storage, then use those compressed representations in the attention computation.

Beyond this, the following techniques are also applied:

• DeepSeekMoE with Auxiliary-Loss-Free Load Balancing (prevents collapse from imbalanced expert assignment)
• Complementary Sequence-Wise Auxiliary Loss (a loss term to address issues introduced by loss-free load balancing)
• Node-Limited Routing (limits communication cost)
• Multi-Token Prediction

---

With that rough overview out of the way, let's get to the main topic: what unusual design choices does R1 actually make?

This is the thing that made me decide to write this post.
There's an odd passage in the DeepSeek-R1 paper.

Fully automated training via RL isn't particularly surprising in the RL domain — but for RL to enable a model to learn "on its own," setting an appropriate policy is critically important.

Reinforcement learning is, at its core:

Training a policy so that an agent, in a given environment, takes actions that maximize reward.

Getting that right is one of the most important and difficult parts of the field.

What makes this notable is that the same thing turns out to be possible in the NLP domain — and that's a genuinely surprising finding.

So how exactly did they implement this?

Think of it this way. Reinforcement learning for an LLM means:

Training a policy — what to generate — so that the generative model, given a prompt as its environment, produces text that maximizes reward.

In other words: if you can define a Reward, you can tune the model so that its generation strategy (Policy) maximizes that reward.

So what is the generation policy, and what serves as the reward?

The optimization method is called GRPO, which looks complex but isn't.

In simple terms, what we want to learn is the policy π. Breaking it down step by step:

D_KL here is the KL-divergence penalty between policies — it measures the difference between two probability distributions.
Put simply, if the two policies diverge too much, they get penalized proportionally.
(It's a regularization term that prevents updates from being too drastic.)

A_i, as you can see from the formula, is a form of Z-score —
it represents the relative reward score of a particular output i.

The objective is based on the probability of a specific sample output appearing under a given policy.

Since the goal is to maximize (current policy / previous policy) * (relative reward score of that output),
the updated policy learns to increase the sampling probability of outputs with higher reward.

The updated model then operates in a way that is consistent with this updated policy.

So what is the Reward — the key value driving this policy update?

Somewhat anticlimactically, the reward model is a simple rule-based system.

Specifically, it uses math problems and coding problems where correctness can be verified programmatically.

On top of that, for reasoning traces, the presence or absence of the special token <think> in the output is also used as a reward signal.

It's a fairly underwhelming reward design, but the paper reports that this approach led to substantial performance improvements in R1-Zero.

Several interesting phenomena were also observed along the way:

• Significant performance gains on the AIME 2024 benchmark (15.6% → 71.0%)
• The emergence of "Aha Moments"

The full R1 model is an evolution of this approach.
To address the limitations of R1-Zero, the authors first did a short warm-up training run on high-quality CoT data, then applied the same RL procedure described above.