XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Bo Zheng +6 more
6/30/2021
cs.CL

Official code: microsoft/unilm (22,168 stars, MIT, last commit 2026-01-23).

Abstract

In this paper, we introduce ELECTRA-style tasks to cross-lingual language model pre-training. Specifically, we present two pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. Besides, we pretrain the model, named as XLM-E, on both multilingual and parallel corpora. Our model outperforms the baseline models on various cross-lingual understanding tasks with much less computation cost. Moreover, analysis shows that XLM-E tends to obtain better cross-lingual transferability.

View on arXivView PDF

Code Implementations(1)

microsoft/unilmOfficial100%
MIT

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

22,1682,703C, C++Jul 23, 20199 months agoMIT
beitbeit-3bitnetdeepnetdocument-ai+15 more

Cite this paper

@article{chi2021xlme,
  title  = {XLM-E: Cross-lingual Language Model Pre-training via ELECTRA},
  author = {Zewen Chi and Shaohan Huang and Li Dong and Shuming Ma and Bo Zheng and Saksham Singhal and Payal Bajaj and Xia Song and Xian-Ling Mao and Heyan Huang and Furu Wei},
  year   = {2021},
  eprint = {2106.16138},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2106.16138}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2106.16138.svg)](https://codeofpaper.com/paper/2106.16138)

Discussion