The topic has a lot of prerequisites to cover in depth, so let's just hit the highlights.
This is a paper from Microsoft, accepted at ICLR 2023. [Paper]
Section 3 of the paper calls out exactly two distinguishing contributions:
1. DISENTANGLED ATTENTION
2. ENHANCED MASK DECODER
Let's take a quick look at each.
DISENTANGLED ATTENTION
To understand why this is done, you need a solid grasp of the Transformer architecture.
Transformers already account for position to a reasonable degree — apparently just not enough.
The key concept here is Position.
What is the Transformer's main advantage?
Information that RNNs had to process sequentially can now be represented as a matrix.
But each row of that matrix is just an embedding vector — it carries no positional information whatsoever.
Position information is critically important, for an obvious reason:
The same word (token) can mean something different depending on whether it appears at the beginning or the end of a sentence.
Vaswani, A. (2017). Attention is all you need. Advances in Neural Information Processing Systems.
That is why Transformers use Positional Encoding — a mechanism that adds positional values directly to the input embeddings.
In practice this is done via Sinusoidal Positional Encoding. Put simply: there are as many wave functions as there are dimensions in D_model, each with a different period spaced at regular intervals. The positional information is encoded through the spatial trace of these phase-shifted wave functions across individual dimensions.
Visualized, it looks like this. This topic alone could fill an entire day, so let's move on.
Sinusoidal Positional Encoding is quite powerful, but at the end of the day it is just a mechanism that adds a fixed set of values to the input.
DeBERTa's Disentangled Attention goes further — it explicitly accounts for Position in addition to Content.
Cross Attention Score
The core idea is straightforward: the attention score is decomposed into four components.
Here, H denotes the Content vector and P denotes the Position vector.
H comes from token embeddings. P is computed based on the relative position between tokens i and j, using the formula below.
The distance between i and j is expressed this way.
In the formula above, K is the maximum relative distance — essentially a threshold that controls how far apart two tokens can be before their relationship is no longer captured.
Multiplying by the weight matrix gives us ...
These are then multiplied by weight matrices to produce the actual Q, K, and V, which are combined as follows to compute the final attention score.
In other words, the standard attention score formula is only slightly modified.
This approach effectively fuses three types of information. (Four components appear in the formulation, but position-to-position is excluded as it carries no meaningful signal.)
Large Language Models: DeBERTa — Decoding-Enhanced BERT with Disentangled Attention (https://towardsdatascience.com/large-language-models-deberta-decoding-enhanced-bert-with-disentangled-attention-90016668db4b)
There is an excellent diagram that captures this perfectly, so I'm reproducing it here.
The input is H, the token embedding matrix. P is the matrix computed in the way described above. The output is H_0, produced by the attention mechanism.
The first step — computing Q_c and K_c, then multiplying by V_c — is what a standard Transformer does. (There is a minor difference in the scaling constant compared to standard Scaled Dot-Product Attention.) Everything else is what DeBERTa adds on top.
ENHANCED MASK DECODER
Decoder comparison
The difference is not dramatic. The key thing to notice is that in a standard decoder, Q, K, and V are all derived from H (the input hidden state), and that is what gets fed into each layer.
In the Enhanced Mask Decoder (EMD), only Q is sourced from a different value (I).
Why?
Recent language models are trained with Masked Language Modeling (MLM): tokens are randomly replaced with a [MASK] token, and the model learns to predict them from the surrounding context.
The problem is that Disentangled Attention has a fundamental limitation in this setting.
Consider the sentence a new store opened beside the new mall.
If it becomes a new [MASK] opened beside the new [MASK], both [MASK] tokens have the word new immediately preceding them.
Disentangled Attention models only the relative position between tokens — it does not model absolute position the way Sinusoidal Positional Encoding does.
In cases like this one, however, absolute positional information is clearly necessary to disambiguate the two masked tokens.
EMD addresses this by injecting absolute position information into I, which is then used to adjust the value of Q.
The authors view absolute position information as something that, if introduced during the attention computation itself, can interfere with relative position learning. At the same time, they find that including it when modeling actual sentence relationships improves performance.
They also note that I is not limited to carrying only absolute position information — future work may identify other useful signals to encode there.
That covers the two core contributions at a high level. Everything else in DeBERTa is largely the same as a standard Transformer.