Rethink the Effectiveness of Text Data Augmentation: An Empirical Analysis

Zhengxiang Shi, Aldo Lipani
6/13/2023
cs.CLcs.AIcs.LG

No official code found. Best implementation: ShiZhengyan/PowerfulPromptFT (76 stars, MIT, last commit 2024-02-04).

Abstract

In recent years, language models (LMs) have made remarkable progress in advancing the field of natural language processing (NLP). However, the impact of data augmentation (DA) techniques on the fine-tuning (FT) performance of these LMs has been a topic of ongoing debate. In this study, we evaluate the effectiveness of three different FT methods in conjugation with back-translation across an array of 7 diverse NLP tasks, including classification and regression types, covering single-sentence and sentence-pair tasks. Contrary to prior assumptions that DA does not contribute to the enhancement of LMs' FT performance, our findings reveal that continued pre-training on augmented data can effectively improve the FT performance of the downstream tasks. In the most favourable case, continued pre-training improves the performance of FT by more than 10% in the few-shot learning setting. Our finding highlights the potential of DA as a powerful tool for bolstering LMs' performance.

View on arXivView PDF

Code Implementations(1)

ShiZhengyan/PowerfulPromptFTCommunity100%
MIT

[NeurIPS 2023 Main Track] This is the repository for the paper titled "Don’t Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner"

7619Shell, PythonApr 25, 20232 years agoMIT
continue-learningfine-tuningin-context-learninginstruction-tuninglanguage-model+10 more

Similar Papers

Empirical Analysis of Efficient Fine-Tuning Methods for Large Pre-Trained Language Models

83% match2024

Optimising Language Models for Downstream Tasks: A Post-Training Perspective

82% match2025

On Evaluation Protocols for Data Augmentation in a Limited Data Scenario

82% match2024

Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner

82% match2 repos1022026

Downstream bias mitigation is all you need

82% match2024

LLMs vs Established Text Augmentation Techniques for Classification: When do the Benefits Outweight the Costs?

82% match2024

Cite this paper

@article{shi2023rethink,
  title  = {Rethink the Effectiveness of Text Data Augmentation: An Empirical Analysis},
  author = {Zhengxiang Shi and Aldo Lipani},
  year   = {2023},
  eprint = {2306.07664},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.07664v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.07664.svg)](https://codeofpaper.com/paper/2306.07664)

Discussion