Fine-Tuning SentenceTransformer with NLI for Better Embeddings

What Is SentenceTransformer?

SentenceTransformer is a framework that converts sentences into fixed-size vectors. It uses NLI (Natural Language Inference) datasets to learn sentence similarity.

NLI Training Structure

flowchart TD
    A[문장 쌍 입력] --> B[BERT Encoder]
    B --> C[Pooling Layer]
    C --> D[문장 벡터]
    D --> E{Loss Function}
    E -->|Entailment| F[유사]
    E -->|Contradiction| G[비유사]
    E -->|Neutral| H[중립]

Custom SentenceTransformer Implementation

In the 1/6 commit, bert_nli_add_custom_sentencetransformer introduced a custom model:

from sentence_transformers import SentenceTransformer, losses

# 커스텀 BERT 기반 SentenceTransformer
model = SentenceTransformer(modules=[
    custom_bert_model,      # 도메인 사전학습된 BERT
    pooling_layer,          # Mean Pooling
    dense_layer             # 차원 축소
])

# NLI 학습
train_loss = losses.SoftmaxLoss(
    model=model,
    sentence_embedding_dimension=768,
    num_labels=3  # entailment, contradiction, neutral
)

STS (Semantic Textual Similarity) Trainer

The 1/14 commit added an STS trainer. The approach first trains basic similarity with NLI, then fine-tunes on STS data to improve precision.

# STS 학습
sts_loss = losses.CosineSimilarityLoss(model=model)
model.fit(
    train_objectives=[(sts_dataloader, sts_loss)],
    evaluator=sts_evaluator,
    epochs=4,
    warmup_steps=100
)

StableAdamW Optimizer Issue

The 1/9 commit, fix_StableAdamW, addressed a problem with StableAdamW. Using StableAdamW instead of the standard AdamW caused the loss to diverge during early training.

Root cause: the weight decay parameter was set too high.
Fix: reduced weight decay from 0.01 to 0.001.

Training Results

Post-training performance on e-commerce product search:

• Retrieval@10 improved by +12% over the baseline multilingual BERT
• Category classification accuracy improved by +8%