Training a Custom Embedding Model for E-Commerce LLMs

Project Background

prj_ecellm is a project for training an embedding model specialized for the e-commerce domain. It ran from December 2024 through January 2025 — roughly two months, across 42 commits.

Why a Custom Embedding Model

graph TD
    A[일반 임베딩 모델] -->|한계| B[도메인 특화 어휘 부족]
    A -->|한계| C[이커머스 유사도 낮음]
    A -->|한계| D[한국어 성능 부족]

    E[커스텀 임베딩] -->|장점| F[도메인 어휘 반영]
    E -->|장점| G[카테고리 유사도 향상]
    E -->|장점| H[한국어 최적화]

Training Pipeline

1. Data collection: E-commerce product data
2. Tokenizer construction: Domain-specific tokenizer
3. Pre-training: MLM (Masked Language Modeling)
4. Fine-tuning: Sentence Transformer + NLI

Tokenizer Work

E-commerce domain vocabulary was added on top of the existing BERT tokenizer:

# 토크나이저 확장
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-multilingual-cased")
new_tokens = load_ecommerce_vocabulary()
tokenizer.add_tokens(new_tokens)

The 12/26 commits were heavily focused on tokenizer work (del_token, del_local_tokenizer, Tokenizer_tool, etc.).

Adding the MLM Task

The 1/15 commit introduced the Masked Language Modeling task. Pre-training on e-commerce text injected domain knowledge into the model.

Development Timeline

12/19: Project initialization
12/26: Tokenizer construction (6 commits)
12/27~31: Training code development
01/03~07: SentenceTransformer training
01/08~09: StableAdamW optimizer modifications
01/13~14: CrossEncoder + Reranker
01/15~17: STS trainer + MLM Task
01/21~23: Full pipeline revisions complete

42 commits total, roughly five weeks of work.