Embedding Is All RAG Needs: Optimizing RAG Retrieval

Embedding Is All RAG Need: RAG Embedding Optimization Experiments

Overview

In February 2025, this project systematically investigated how embedding model performance affects overall RAG (Retrieval-Augmented Generation) system quality. Despite spanning only 3 commits, it used quantitative comparisons via MTEB (Massive Text Embedding Benchmark) to validate the importance of embedding model selection.

Background

RAG system performance depends on three core components:

1. Retrieval — how well the system finds relevant documents
2. Augmentation — how retrieved documents are incorporated into the prompt
3. Generation — how well the LLM produces a final answer

The embedding model is the backbone of Retrieval, and this project was designed to test the hypothesis that "embedding is all RAG needs."

Commit Timeline

2025-02-25 — first commit (project initialization)
2025-02-25 — benchmark (run MTEB benchmarks)
2025-02-26 — benchmark update (update results)

MTEB-Based Experiments

Models Evaluated

Various embedding models were compared against the MTEB leaderboard:

# Models under evaluation (estimated)
models = [
    "text-embedding-3-large",     # OpenAI
    "text-embedding-3-small",     # OpenAI
    "bge-large-en-v1.5",         # BAAI
    "voyage-large-2",             # Voyage AI
    "e5-mistral-7b-instruct",    # Microsoft
    "gte-Qwen2-7B-instruct",    # Alibaba
]

Evaluation Metrics

Models were evaluated across the range of tasks provided by MTEB:

• Retrieval: document retrieval accuracy (nDCG@10, MAP)
• STS (Semantic Textual Similarity): sentence similarity correlation
• Classification: classification accuracy
• Clustering: clustering V-measure
• Reranking: reranking MAP

Key Findings

1. Model size vs. performance: 7B-parameter models (e.g., e5-mistral) are not always the top performer
2. Importance of domain specialization: a high score on a general-purpose benchmark does not guarantee strong performance in a specific domain, where a smaller model may outperform
3. Cost-performance trade-off: text-embedding-3-small delivers reasonable performance for its cost

Applying the Results

The conclusions from this experiment directly shaped the direction of the subsequent prj_ecellm project (e-commerce-specialized embedding):

• Confirmed the limitations of general-purpose models → derived the need to train a domain-specialized embedding model
• Gained experience with the MTEB evaluation pipeline → reused the same framework in the ecellm project
• Concluded that SentenceTransformer-based fine-tuning is cost-efficient

Retrospective

This was a small project — just 3 commits — but it provided data-driven justification for RAG system architecture decisions. The most lasting outcome was establishing the habit of validating model choices through systematic benchmarking rather than gut instinct.