Better Language Models of Code through Self-Improvement

Hung Quoc To, Nghi D. Q. Bui, Jin Guo, Tien N. Nguyen
4/2/2023
cs.CLcs.AI

No official code found. Best implementation: Fsoft-AIC/Code-LM-Self-Improvement (9 stars, MIT, last commit 2023-05-04).

Abstract

Pre-trained language models for code (PLMCs) have gained attention in recent research. These models are pre-trained on large-scale datasets using multi-modal objectives. However, fine-tuning them requires extensive supervision and is limited by the size of the dataset provided. We aim to improve this issue by proposing a simple data augmentation framework. Our framework utilizes knowledge gained during the pre-training and fine-tuning stage to generate pseudo data, which is then used as training data for the next step. We incorporate this framework into the state-of-the-art language models, such as CodeT5, CodeBERT, and UnixCoder. The results show that our framework significantly improves PLMCs' performance in code-related sequence generation tasks, such as code summarization and code generation in the CodeXGLUE benchmark.

View on arXivView PDF

Code Implementations(1)

Fsoft-AIC/Code-LM-Self-ImprovementCommunity73%
MIT

[ACL 2023] Better Language Models of Code through Self-Improvement

90Shell, PythonMay 4, 20233 years agoMIT

Similar Papers

Spanish Pre-trained BERT Model and Evaluation Data

88% match2 repos5072023

Exploring Data Augmentation for Code Generation Tasks

87% match2023

Kotlin ML Pack: Technical Report

87% match2024

GenCode: A Generic Data Augmentation Framework for Boosting Deep Learning-Based Code Understanding

87% match1 repo12024

Code Representation Pre-training with Complements from Program Executions

85% match2023

PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback

85% match2023

Cite this paper

@article{to2023better,
  title  = {Better Language Models of Code through Self-Improvement},
  author = {Hung Quoc To and Nghi D. Q. Bui and Jin Guo and Tien N. Nguyen},
  year   = {2023},
  eprint = {2304.01228},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.01228v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.01228.svg)](https://codeofpaper.com/paper/2304.01228)

Discussion