A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning

Yo Joong Choe, Jiyeon Ham, Kyubyong Park, Yeoil Yoon
7/2/2019
cs.CLcs.LG

Official code: kakaobrain/helo-word (93 stars, Apache-2.0, last commit 2019-09-19).

Abstract

Grammatical error correction can be viewed as a low-resource sequence-to-sequence task, because publicly available parallel corpora are limited. To tackle this challenge, we first generate erroneous versions of large unannotated corpora using a realistic noising function. The resulting parallel corpora are subsequently used to pre-train Transformer models. Then, by sequentially applying transfer learning, we adapt these models to the domain and style of the test set. Combined with a context-aware neural spellchecker, our system achieves competitive results in both restricted and low resource tracks in ACL 2019 BEA Shared Task. We release all of our code and materials for reproducibility.

View on arXivView PDF

Code Implementations(1)

kakaobrain/helo-wordOfficial100%
Apache-2.0

Team Kakao&Brain's Grammatical Error Correction System for the ACL 2019 BEA Shared Task

9323C++, LuaAug 12, 20197 years agoApache-2.0
deep-learningfairseqgrammatical-error-correctionnlppre-training+2 more

Similar Papers

Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule

84% match2023

Byte-Level Grammatical Error Correction Using Synthetic and Curated Corpora

82% match1 repo82023

Mitigating Data Scarcity for Large Language Models

82% match2023

Beqi: Revitalize the Senegalese Wolof Language with a Robust Spelling Corrector

82% match2023

Multilingual Denoising Pre-training for Neural Machine Translation

81% match1 repo32.2k2020

An Error-Guided Correction Model for Chinese Spelling Error Correction

81% match1 repo142023

Cite this paper

@article{choe2019neural,
  title  = {A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning},
  author = {Yo Joong Choe and Jiyeon Ham and Kyubyong Park and Yeoil Yoon},
  year   = {2019},
  eprint = {1907.01256},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1907.01256}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1907.01256.svg)](https://codeofpaper.com/paper/1907.01256)

Discussion