Improving Sentence Embeddings with Automatic Generation of Training Data Using Few-shot Examples

Soma Sato, Hayato Tsukagoshi, Ryohei Sasano, Koichi Takeda
2/23/2024
cs.CLcs.LG

No official code found. Best implementation: lamsoma/Auto_NLI (1 star, last commit 2025-02-12).

Abstract

Decoder-based large language models (LLMs) have shown high performance on many tasks in natural language processing. This is also true for sentence embedding learning, where a decoder-based model, PromptEOL, has achieved the best performance on semantic textual similarity (STS) tasks. However, PromptEOL requires a manually annotated natural language inference (NLI) dataset for fine-tuning. We aim to improve sentence embeddings without using large manually annotated datasets by automatically generating an NLI dataset with an LLM and using it for fine-tuning of PromptEOL. To achieve this, we explore methods of data generation suitable for sentence embedding learning in this study. Specifically, we will focus on automatic dataset generation through few-shot learning and explore the appropriate methods to leverage few-shot examples. Experimental results on the STS tasks demonstrate that our approach outperforms existing models in settings without large manually annotated datasets.

View on arXivView PDF

Code Implementations(1)

lamsoma/Auto_NLICommunity100%
10Shell, PythonJan 5, 20241 years ago

Similar Papers

Simple Techniques for Enhancing Sentence Embeddings in Generative Language Models

86% match1 repo262024

Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Models

86% match1 repo832025

Making Text Embedders Few-Shot Learners

85% match1 repo11.9k2024

Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs

85% match2024

Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning

85% match2024

Optimizing Sentence Embedding with Pseudo-Labeling and Model Ensembles: A Hierarchical Framework for Enhanced NLP Tasks

84% match2025

Cite this paper

@article{sato2024improving,
  title  = {Improving Sentence Embeddings with Automatic Generation of Training Data Using Few-shot Examples},
  author = {Soma Sato and Hayato Tsukagoshi and Ryohei Sasano and Koichi Takeda},
  year   = {2024},
  eprint = {2402.15132},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.15132v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.15132.svg)](https://codeofpaper.com/paper/2402.15132)

Discussion