Distilling Knowledge Learned in BERT for Text Generation

Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, Jingjing Liu
11/10/2019
cs.CLcs.LG

Official code: ChenRocks/Distill-BERT-Textgen (128 stars, MIT, last commit 2021-06-30).

Abstract

Large-scale pre-trained language model such as BERT has achieved great success in language understanding tasks. However, it remains an open question how to utilize BERT for language generation. In this paper, we present a novel approach, Conditional Masked Language Modeling (C-MLM), to enable the finetuning of BERT on target generation tasks. The finetuned BERT (teacher) is exploited as extra supervision to improve conventional Seq2Seq models (student) for better text generation performance. By leveraging BERT's idiosyncratic bidirectional nature, distilling knowledge learned in BERT can encourage auto-regressive Seq2Seq models to plan ahead, imposing global sequence-level supervision for coherent text generation. Experiments show that the proposed approach significantly outperforms strong Transformer baselines on multiple language generation tasks such as machine translation and text summarization. Our proposed model also achieves new state of the art on IWSLT German-English and English-Vietnamese MT datasets. Code is available at https://github.com/ChenRocks/Distill-BERT-Textgen.

View on arXivView PDF

Code Implementations(1)

ChenRocks/Distill-BERT-TextgenOfficial100%
MIT

Research code for ACL 2020 paper: "Distilling Knowledge Learned in BERT for Text Generation".

12820Shell, PythonApr 17, 20205 years agoMIT
bert-modelknowledge-distillationmachine-translationnatural-language-processing

Similar Papers

MVP: Multi-task Supervised Pre-training for Natural Language Generation

88% match2 repos162.1k2022

Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models

87% match1 repo7.5k2023

Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages

86% match2023

Teach LLMs to Personalize -- An Approach inspired by Writing Education

86% match2023

Leveraging Natural Supervision for Language Representation Learning and Generation

85% match1 repo802022

FreeLM: Fine-Tuning-Free Language Model

85% match2023

Cite this paper

@article{chen2019distilling,
  title  = {Distilling Knowledge Learned in BERT for Text Generation},
  author = {Yen-Chun Chen and Zhe Gan and Yu Cheng and Jingzhou Liu and Jingjing Liu},
  year   = {2019},
  eprint = {1911.03829},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1911.03829}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1911.03829.svg)](https://codeofpaper.com/paper/1911.03829)

Discussion