Improving Contrastive Learning of Sentence Embeddings with Focal-InfoNCE

Pengyue Hou, Xingyu Li
10/10/2023
cs.CLcs.LG

Official code: puerrrr/Focal-InfoNCE (8 stars, MIT, last commit 2023-08-25).

Abstract

The recent success of SimCSE has greatly advanced state-of-the-art sentence representations. However, the original formulation of SimCSE does not fully exploit the potential of hard negative samples in contrastive learning. This study introduces an unsupervised contrastive learning framework that combines SimCSE with hard negative mining, aiming to enhance the quality of sentence embeddings. The proposed focal-InfoNCE function introduces self-paced modulation terms in the contrastive objective, downweighting the loss associated with easy negatives and encouraging the model focusing on hard negatives. Experimentation on various STS benchmarks shows that our method improves sentence embeddings in terms of Spearman's correlation and representation alignment and uniformity.

View on arXivView PDF

Code Implementations(1)

puerrrr/Focal-InfoNCEOfficial100%
MIT
80Shell, PythonJun 14, 20233 years agoMIT

Similar Papers

HNCSE: Advancing Sentence Embeddings via Hybrid Contrastive Learning with Hard Negatives

86% match2024

Instance Smoothed Contrastive Learning for Unsupervised Sentence Embedding

83% match1 repo92023

KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learning

83% match1 repo102024

Clustering-Aware Negative Sampling for Unsupervised Sentence Representation

83% match1 repo132023

Self-Adaptive Reconstruction with Contrastive Learning for Unsupervised Sentence Embeddings

82% match1 repo2024

Non-contrastive sentence representations via self-supervision

82% match2023

Cite this paper

@article{hou2023improving,
  title  = {Improving Contrastive Learning of Sentence Embeddings with Focal-InfoNCE},
  author = {Pengyue Hou and Xingyu Li},
  year   = {2023},
  eprint = {2310.06918},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.06918v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.06918.svg)](https://codeofpaper.com/paper/2310.06918)

Discussion