Improving Text Embeddings for Smaller Language Models Using Contrastive Fine-tuning

Trapoom Ukarapol, Zhicheng Lee, Amy Xin
8/1/2024
cs.CL

Official code: trapoom555/Language-Model-STS-CFT (64 stars, last commit 2024-08-02).

Abstract

While Large Language Models show remarkable performance in natural language understanding, their resource-intensive nature makes them less accessible. In contrast, smaller language models such as MiniCPM offer more sustainable scalability, but often underperform without specialized optimization. In this paper, we explore the enhancement of smaller language models through the improvement of their text embeddings. We select three language models, MiniCPM, Phi-2, and Gemma, to conduct contrastive fine-tuning on the NLI dataset. Our results demonstrate that this fine-tuning method enhances the quality of text embeddings for all three models across various benchmarks, with MiniCPM showing the most significant improvements of an average 56.33% performance gain. The contrastive fine-tuning code is publicly available at https://github.com/trapoom555/Language-Model-STS-CFT.

View on arXivView PDF

Code Implementations(1)

trapoom555/Language-Model-STS-CFTOfficial100%

Improving Text Embedding of Language Models Using Contrastive Fine-tuning

649Shell, PythonApr 16, 20242 years ago

Similar Papers

Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe

86% match1 repo32024

Resource-Efficient Adaptation of Large Language Models for Text Embeddings via Prompt Engineering and Contrastive Fine-tuning

86% match2025

Bootstrapping Embeddings for Low Resource Languages

85% match2026

From Superficial Patterns to Semantic Understanding: Fine-Tuning Language Models on Contrast Sets

85% match2025

Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models

84% match2025

Small Language Models Improve Giants by Rewriting Their Outputs

83% match1 repo122023

Cite this paper

@article{ukarapol2024improving,
  title  = {Improving Text Embeddings for Smaller Language Models Using Contrastive Fine-tuning},
  author = {Trapoom Ukarapol and Zhicheng Lee and Amy Xin},
  year   = {2024},
  eprint = {2408.00690},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2408.00690v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2408.00690.svg)](https://codeofpaper.com/paper/2408.00690)

Discussion