Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation

Jian Guan, Minlie Huang
7/4/2023
cs.CL

No official code found. Best implementation: thu-coai/SelfCont (4 stars, last commit 2023-07-06).

Abstract

Despite the huge progress in myriad generation tasks, pretrained language models (LMs) such as GPT2 still tend to generate repetitive texts with maximization-based decoding algorithms for open-ended generation. We attribute their overestimation of token-level repetition probabilities to the learning bias: LMs capture simple repetitive patterns faster with the MLE loss. We propose self-contrastive training to penalize the output of a premature checkpoint of the same model when it incorrectly predicts repetition, which is shown to mitigate repetition effectively while maintaining fluency on two datasets. Furthermore, we find that LMs use longer-range dependencies to predict repetitive tokens than non-repetitive ones, which may be the cause of sentence-level repetition loops.

View on arXivView PDF

Code Implementations(1)

thu-coai/SelfContCommunity100%

Code for the paper "Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation"

40Shell, PythonMay 9, 20233 years ago

Similar Papers

MiLe Loss: a New Entropy-Weighed Loss for Mitigating the Bias of Learning Difficulties in Large Language Models

86% match1 repo692023

Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability

86% match1 repo12023

Uncovering Uncontrolled Repetition through Residual Stream Dynamics

86% match2026

FOCUS & RePAIR: Mitigating Text Degeneration via Token-Level Guidance for Pruned Large Language Models

85% match2026

BitLM: Unlocking Multi-Token Language Generation with Bitwise Continuous Diffusion

85% match2026

Internal Data Repetition Destroys Language Models

85% match2026

Cite this paper

@article{guan2023mitigating,
  title  = {Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation},
  author = {Jian Guan and Minlie Huang},
  year   = {2023},
  eprint = {2307.01542},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2307.01542v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2307.01542.svg)](https://codeofpaper.com/paper/2307.01542)

Discussion