ConLID: Supervised Contrastive Learning for Low-Resource Language Identification

Negar Foroutan, Jakhongir Saydaliev, Ye Eun Kim, Antoine Bosselut
6/18/2025
cs.CLcs.AIcs.LG

No official code found. Best implementation: epfl-nlp/ConLID (8 stars, last commit 2025-09-27).

Abstract

Language identification (LID) is a critical step in curating multilingual LLM pretraining corpora from web crawls. While many studies on LID model training focus on collecting diverse training data to improve performance, low-resource languages -- often limited to single-domain data, such as the Bible -- continue to perform poorly. To resolve these class imbalance and bias issues, we propose a novel supervised contrastive learning (SCL) approach to learn domain-invariant representations for low-resource languages. Through an extensive analysis, we show that our approach improves LID performance on out-of-domain data for low-resource languages by 3.2%, demonstrating its effectiveness in enhancing LID models.

View on arXivView PDF

Code Implementations(1)

epfl-nlp/ConLIDCommunity64%

ConLID: Supervised Contrastive Learning for Low-Resource Language Identification [EACL - 2026]

82Shell, PythonMay 23, 20251 years ago
language-identificationlow-resource-languagesmultilingual-language-modelsmultilingual-nlp

Similar Papers

It's All About In-Context Learning! Teaching Extremely Low-Resource Languages to LLMs

84% match2025

Unsupervised Data Validation Methods for Efficient Model Training

83% match2024

Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification

83% match2026

What Language is This? Ask Your Tokenizer

83% match2026

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

82% match2026

A Rigorous Evaluation of LLM Data Generation Strategies for Low-Resource Languages

82% match2025

Cite this paper

@article{foroutan2025conlid,
  title  = {ConLID: Supervised Contrastive Learning for Low-Resource Language Identification},
  author = {Negar Foroutan and Jakhongir Saydaliev and Ye Eun Kim and Antoine Bosselut},
  year   = {2025},
  eprint = {2506.15304},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.15304v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.15304.svg)](https://codeofpaper.com/paper/2506.15304)

Discussion