PMIndia -- A Collection of Parallel Corpora of Languages of India

Barry Haddow, Faheem Kirefu
1/27/2020
cs.CL

Official code: bhaddow/pmindia-crawler (17 stars, MIT, last commit 2020-01-28).

Abstract

Parallel text is required for building high-quality machine translation (MT) systems, as well as for other multilingual NLP applications. For many South Asian languages, such data is in short supply. In this paper, we described a new publicly available corpus (PMIndia) consisting of parallel sentences which pair 13 major languages of India with English. The corpus includes up to 56000 sentences for each language pair. We explain how the corpus was constructed, including an assessment of two different automatic sentence alignment methods, and present some initial NMT results on the corpus.

View on arXivView PDF

Code Implementations(1)

bhaddow/pmindia-crawlerOfficial100%
MIT

Code for extracting parallel corpora from pmindia

171Shell, PythonDec 17, 20196 years agoMIT

Similar Papers

NAIST-SIC-Aligned: an Aligned English-Japanese Simultaneous Interpretation Corpus

88% match1 repo32023

Building a Parallel Corpus and Training Translation Models Between Luganda and English

87% match2023

SAHAAYAK 2023 -- the Multi Domain Bilingual Parallel Corpus of Sanskrit to Hindi for Machine Translation

87% match2023

Bilingual Corpus Mining and Multistage Fine-Tuning for Improving Machine Translation of Lecture Transcripts

87% match1 repo152023

WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia

86% match1 repo3.7k2019

Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond

85% match1 repo3.7k2018

Cite this paper

@article{haddow2020pmindia,
  title  = {PMIndia -- A Collection of Parallel Corpora of Languages of India},
  author = {Barry Haddow and Faheem Kirefu},
  year   = {2020},
  eprint = {2001.09907},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2001.09907}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2001.09907.svg)](https://codeofpaper.com/paper/2001.09907)

Discussion