Quality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora

Surangika Ranathunga, Nisansa de Silva, Menan Velayuthan, Aloka Fernando, Charitha Rathnayake
2/12/2024
cs.CL

No confident code implementation found for this paper.

Abstract

We conducted a detailed analysis on the quality of web-mined corpora for two low-resource languages (making three language pairs, English-Sinhala, English-Tamil and Sinhala-Tamil). We ranked each corpus according to a similarity measure and carried out an intrinsic and extrinsic evaluation on different portions of this ranked corpus. We show that there are significant quality differences between different portions of web-mined corpora and that the quality varies across languages and datasets. We also show that, for some web-mined datasets, Neural Machine Translation (NMT) models trained with their highest-ranked 25k portion can be on par with human-curated datasets.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Do Language Models Care About Text Quality? Evaluating Web-Crawled Corpora Across 11 Languages

86% match2024

Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation

85% match2024

Parallel Corpora for Machine Translation in Low-resource Indic Languages: A Comprehensive Review

84% match2025

Improving the quality of Web-mined Parallel Corpora of Low-Resource Languages using Debiasing Heuristics

83% match2025

A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism

82% match1 repo92024

Developing an English-Efik Corpus and Machine Translation System for Digitization Inclusion

82% match2026

Cite this paper

@article{ranathunga2024quality,
  title  = {Quality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora},
  author = {Surangika Ranathunga and Nisansa de Silva and Menan Velayuthan and Aloka Fernando and Charitha Rathnayake},
  year   = {2024},
  eprint = {2402.07446},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2402.07446v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2402.07446.svg)](https://codeofpaper.com/paper/2402.07446)

Discussion