Refining Joint Text and Source Code Embeddings for Retrieval Task with Parameter-Efficient Fine-Tuning

Karim Galliamov, Leila Khaertdinova, Karina Denisova
5/7/2024
cs.LGcs.SE

No official code found. Best implementation: leiluk1/CodeSearcher (4 stars, last commit 2024-05-23).

Abstract

The latest developments in Natural Language Processing (NLP) have demonstrated remarkable progress in a code-text retrieval problem. As the Transformer-based models used in this task continue to increase in size, the computational costs and time required for end-to-end fine-tuning become substantial. This poses a significant challenge for adapting and utilizing these models when computational resources are limited. Motivated by these concerns, we propose a fine-tuning framework that leverages Parameter-Efficient Fine-Tuning (PEFT) techniques. Moreover, we adopt contrastive learning objectives to improve the quality of bimodal representations learned by transformer models. Additionally, for PEFT methods we provide extensive benchmarking, the lack of which has been highlighted as a crucial problem in the literature. Based on the thorough experimentation with the CodeT5+ model conducted on two datasets, we demonstrate that the proposed fine-tuning framework has the potential to improve code-text retrieval performance by tuning only 0.4% parameters at most.

View on arXivView PDF

Code Implementations(1)

leiluk1/CodeSearcherCommunity100%

Solving the task of finding source code given its description, which could be useful for both experienced and novice programmers

40Python, Jupyter NotebookSep 17, 20232 years ago
codesearchcodet5deep-learningpeft

Similar Papers

Efficient fine-tuning methodology of text embedding models for information retrieval: contrastive learning penalty (clp)

84% match1 repo112024

CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval

82% match2024

GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning

80% match2024

Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval

80% match1 repo32024

Improving embedding with contrastive fine-tuning on small datasets with expert-augmented scores

80% match2024

Towards General Text Embeddings with Multi-stage Contrastive Learning

80% match2023

Cite this paper

@article{galliamov2024refining,
  title  = {Refining Joint Text and Source Code Embeddings for Retrieval Task with Parameter-Efficient Fine-Tuning},
  author = {Karim Galliamov and Leila Khaertdinova and Karina Denisova},
  year   = {2024},
  eprint = {2405.04126},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2405.04126v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2405.04126.svg)](https://codeofpaper.com/paper/2405.04126)

Discussion