Embedding Models and Sentence-Transformer Training
17 min readFeb 21, 2025Feb 21, 2026
Dedicated embedding models ...
Embeddings are a critical area of NLP.
Whether you're working with LLMs or sLLMs, computers need some kind of transformation process to handle natural language.
We call this process of mapping text or sentences into a latent space embedding.
Through this process, a computer converts sentences into numbers that it can use in computation.
With the right latent space, you can also determine which sentences end up in similar regions of that space.
Because it's useful in so many ways, embeddings have a wide range of applications.
As LLM capabilities have advanced dramatically and RAG adoption has grown, interest in embeddings has surged.
The key application areas are:
1. Sentence Similarity
2. Sentence Classification
3. Sentence Reranking
4. Sentence Retrieval
These categories are distinct on the surface, but at the end of the day they all come down to the same thing.
It's all about evaluating whether an embedded vector is a good representation.
(There are approaches like Cross-Encoding that feel a bit different, but let's skip that for now.)
Sentence similarity is computed as the distance between embedding vectors,
embedding vectors determine the class in classification,
and embedding vectors are used to rank and retrieve similar sentences.
What this post covers is how to actually build such embedding models.
1. Pre-Training
This stage can be skipped.
Well-trained pre-trained models are widely available on Hugging Face.
The main thing to consider is whether the model can handle your target language (e.g., Korean) adequately.
If you want to train from scratch, you can run a large corpus through a task like MLM.
Data quality is critically important here.
After extensive experimentation, the community has converged on the following:
1. Books produce very high-quality training text.
2. News articles also yield good-quality text.
3. Cleaned Wikipedia data is decent as well.
4. Sources like Namuwiki or comment data are harder for the model to learn from. (In my case, gradient explosion kept occurring.)
5. Other unfiltered, randomly crawled data — breaks things intermittently.
The problem is that books are, by default, copyrighted material,
and news articles are also technically copyrighted.
Beyond the licensing issues, just finding clean Korean text in the first place is genuinely hard.
(There's a wealth of good English data by comparison.)
Regardless, collect what you can and run MLM pre-training. That's the setup done.
*(Always remember: Trash In, Trash Out.)
2. Down-Stream Task: Data Pre-Processing
This is where it gets daunting.
Not quite hellish, but definitely where the sense of "where do I even start" kicks in.
The four core embedding tasks we discussed earlier don't just fall into place by training on MLM alone.
(Classification scores are suspiciously high ...) MTEB benchmark scores for KoModernBERT-base, trained only on an MLM task.
Take this example.
A ModernBERT model fine-tuned on Korean data currently sits at rank 13 on the leaderboard.
(Compared against larger models, the gap is naturally even bigger.)
To push this further, the model needs to be trained on a variety of tasks.
Some key ones are:
1. Distinguishing entailment, contradiction, and neutral relationships between sentences (Natural Language Inference; NLI)
2. Estimating the semantic similarity between sentences (Semantic Textual Similarity; STS)
3. Given a query, finding similar sentences (Positives) and distinguishing dissimilar ones (Negatives)
These are the main directions worth considering.
The difficulty is that these downstream fine-tuning tasks can substantially improve real-world embedding performance,
yet Korean-language training data for them is extremely scarce.
Searching for Korean triplet datasets on Hugging Face — the kind essential for these tasks — returns only 3 results.
DeepSeek's recent R1 training approach applies reinforcement learning to boost performance.
The takeaway is that performance really comes through when actions receive appropriate rewards.
That's not the main point here, though — I'm raising it to contextualize the current trend.
Anyone who has spent time reading embedding research will have encountered all of these.
Looking at what they have in common, you'll notice they share a similar loss function structure.
They all feel like variations on the same theme ...
The details differ, but the goal is the same across all of them.
Train the model to pull representations closer to preferred answers and push them away from non-preferred ones.
In practice, this approach consistently outperforms training only on positive examples.
3. Loss Function
Coming back to the main point — what I really wanted to write about is Sentence Transformers' loss functions.
To properly teach a model [the patterns in data], you need a well-designed [loss function] for that purpose.
As we've repeatedly seen in the papers above, the right loss function plays a central role.
Pairing the right data with the right loss function is therefore critical —
and Sentence Transformers has solid implementations of all the relevant losses.
And more ...
The common thread across all these losses is that rather than training on positives alone,
they construct negatives in some form and train the model to move away from them.
We won't go into the detailed math here — let's focus on the concepts.
It's a distance problem in its simplest form: pull the anchor closer to the positive and push it away from the negative.
Straightforward in concept.
MultipleNegativesRankingLoss
Here the objective shifts to minimizing a log-likelihood.
Everything except the correct answer I is a randomly sampled negative.
The goal is to minimize the contribution of these negatives.
MultipleNegativesSymmetricRankingLoss
Given a list of (anchor, positive) pairs, this loss sums the following two losses:
Forward loss: Given an anchor, find the sample with the highest similarity out of all positives in the batch. This is equivalent to MultipleNegativesRankingLoss.
Backward loss: Given a positive, find the sample with the highest similarity out of all anchors in the batch.
This is a variant of the previous loss.
Everything so far has used [Positives] and [Negatives] with respect to an input [sentence].
In other words, the objective has been: given an input sentence, find the right [Positive].
But then — given a [Positive], which input sentence does it best correspond to? This loss reinforces exactly that.
It additionally applies the same mechanism in reverse: for a [Positive], use the matching [sentence] as the positive and treat all other [sentences] as negatives.
This further strengthens the [sentence] — [Positive] relationship.
GISTEmbedLoss
But is it actually correct to treat all other in-batch samples as negatives?
With random sampling, there's a real chance that some of those "negatives" aren't true negatives at all.
GIST (Guided In-sample Selection of Training Negatives) is a method that addresses this.
True to its name, it uses a guide model to verify whether a randomly sampled negative is actually a true negative.
This produces more precise triplet-style training.
That's a quick overview of the loss functions available in Sentence Transformers.
In practice, multi-negative approaches outperform simple triplet loss by a significant margin,
and larger in-batch negative sets are known to yield better results.
(To push in-batch size as large as possible, cached loss functions are typically used.)
Accordingly, Sentence Transformers includes many cached variants of these loss functions built in.
They're not too difficult to understand, so read through the docs and pick what fits your use case.
