IndicBART: A Pre-trained Model for Indic Natural Language Generation

Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan, Ratish Puduppully, Mitesh M. Khapra +1 more
9/7/2021
cs.CLcs.AI

Official code: AI4Bharat/indic-bart (52 stars, MIT, last commit 2022-07-20).

Abstract

In this paper, we study pre-trained sequence-to-sequence models for a group of related languages, with a focus on Indic languages. We present IndicBART, a multilingual, sequence-to-sequence pre-trained model focusing on 11 Indic languages and English. IndicBART utilizes the orthographic similarity between Indic scripts to improve transfer learning between similar Indic languages. We evaluate IndicBART on two NLG tasks: Neural Machine Translation (NMT) and extreme summarization. Our experiments on NMT and extreme summarization show that a model specific to related languages like IndicBART is competitive with large pre-trained models like mBART50 despite being significantly smaller. It also performs well on very low-resource translation scenarios where languages are not included in pre-training or fine-tuning. Script sharing, multilingual training, and better utilization of limited model capacity contribute to the good performance of the compact IndicBART model.

View on arXivView PDF

Code Implementations(1)

AI4Bharat/indic-bartOfficial100%
MIT

Pre-trained, multilingual sequence-to-sequence models for Indian languages

525PythonSep 6, 20214 years agoMIT

Similar Papers

Sequence to sequence pretraining for a less-resourced Slovenian language

87% match2022

Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq

87% match1 repo1.6k2018

PEACH: Pre-Training Sequence-to-Sequence Multilingual Models for Translation with Semi-Supervised Pseudo-Parallel Document Generation

86% match1 repo32023

Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning

86% match1 repo302023

Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages

86% match2023

Improving Polish to English Neural Machine Translation with Transfer Learning: Effects of Data Volume and Language Similarity

86% match2023

Cite this paper

@article{dabre2021indicbart,
  title  = {IndicBART: A Pre-trained Model for Indic Natural Language Generation},
  author = {Raj Dabre and Himani Shrotriya and Anoop Kunchukuttan and Ratish Puduppully and Mitesh M. Khapra and Pratyush Kumar},
  year   = {2021},
  eprint = {2109.02903},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2109.02903}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2109.02903.svg)](https://codeofpaper.com/paper/2109.02903)

Discussion