Efficient fine-tuning methodology of text embedding models for information retrieval: contrastive learning penalty (clp)

Jeongsu Yu
12/23/2024
cs.IRcs.AI

Official code: CreaLabs/Enhanced-BGE-M3-with-CLP-and-MoE (11 stars, last commit 2024-12-27).

Abstract

Text embedding models play a crucial role in natural language processing, particularly in information retrieval, and their importance is further highlighted with the recent utilization of RAG (Retrieval- Augmented Generation). This study presents an efficient fine-tuning methodology encompassing data selection, loss function, and model architecture to enhance the information retrieval performance of pre-trained text embedding models. In particular, this study proposes a novel Contrastive Learning Penalty function that overcomes the limitations of existing Contrastive Learning. The proposed methodology achieves significant performance improvements over existing methods in document retrieval tasks. This study is expected to contribute to improving the performance of information retrieval systems through fine-tuning of text embedding models. The code for this study can be found at https://github.com/CreaLabs/Enhanced-BGE-M3-with-CLP-and-MoE, and the best-performing model can be found at https://huggingface.co/CreaLabs.

View on arXivView PDF

Code Implementations(1)

CreaLabs/Enhanced-BGE-M3-with-CLP-and-MoEOfficial100%

This repository provides the code for applying Contrastive Learning Penalty Loss (CLPL) and Mixture of Experts (MoE) to the BGE-M3 text embedding model for enhanced information retrieval performance.

110PythonNov 27, 20241 years ago

Similar Papers

Improving embedding with contrastive fine-tuning on small datasets with expert-augmented scores

85% match2024

Refining Joint Text and Source Code Embeddings for Retrieval Task with Parameter-Efficient Fine-Tuning

84% match1 repo42024

Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe

83% match1 repo32024

jina-embeddings-v5-text: Task-Targeted Embedding Distillation

82% match2026

HEAL: Hierarchical Embedding Alignment Loss for Improved Retrieval and Representation Learning

82% match1 repo262024

Improving Embedding Accuracy for Document Retrieval Using Entity Relationship Maps and Model-Aware Contrastive Sampling

82% match2024

Cite this paper

@article{yu2024efficient,
  title  = {Efficient fine-tuning methodology of text embedding models for information retrieval: contrastive learning penalty (clp)},
  author = {Jeongsu Yu},
  year   = {2024},
  eprint = {2412.17364},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.17364v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.17364.svg)](https://codeofpaper.com/paper/2412.17364)

Discussion