Teaching the Pre-trained Model to Generate Simple Texts for Text Simplification

Renliang Sun, Wei Xu, Xiaojun Wan
5/21/2023
cs.CLcs.AI

No official code found. Best implementation: RLSNLP/SimpleBART (4 stars, MIT, last commit 2023-06-15).

Abstract

Randomly masking text spans in ordinary texts in the pre-training stage hardly allows models to acquire the ability to generate simple texts. It can hurt the performance of pre-trained models on text simplification tasks. In this paper, we propose a new continued pre-training strategy to teach the pre-trained model to generate simple texts. We continue pre-training BART, a representative model, to obtain SimpleBART. It consistently and significantly improves the results on lexical simplification, sentence simplification, and document-level simplification tasks over BART. At the end, we compare SimpleBART with several representative large language models (LLMs).

View on arXivView PDF

Code Implementations(1)

RLSNLP/SimpleBARTCommunity100%
MIT
41PythonMay 16, 20233 years agoMIT

Similar Papers

Frequency Is What You Need: Considering Word Frequency When Text Masking Benefits Vision-Language Model Pre-training

85% match1 repo22024

Right at My Level: A Unified Multilingual Framework for Proficiency-Aware Text Simplification

85% match2026

EvoText: Enhancing Natural Language Generation Models via Self-Escalation Learning for Up-to-Date Knowledge and Improved Performance

85% match2023

Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning

85% match2026

Knowledge Editing for Large Language Models: A Survey

85% match1 repo1.2k2023

Generate to Understand for Representation

84% match1 repo142023

Cite this paper

@article{sun2023teaching,
  title  = {Teaching the Pre-trained Model to Generate Simple Texts for Text Simplification},
  author = {Renliang Sun and Wei Xu and Xiaojun Wan},
  year   = {2023},
  eprint = {2305.12463},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2305.12463v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2305.12463.svg)](https://codeofpaper.com/paper/2305.12463)

Discussion