L3Cube-IndicNews: News-based Short Text and Long Document Classification Datasets in Indic Languages

Aishwarya Mirashi, Srushti Sonavane, Purva Lingayat, Tejas Padhiyar, Raviraj Joshi
1/4/2024
cs.CLcs.LG

Official code: l3cube-pune/indic-nlp (23 stars, last commit 2025-06-07).

Abstract

In this work, we introduce L3Cube-IndicNews, a multilingual text classification corpus aimed at curating a high-quality dataset for Indian regional languages, with a specific focus on news headlines and articles. We have centered our work on 10 prominent Indic languages, including Hindi, Bengali, Marathi, Telugu, Tamil, Gujarati, Kannada, Odia, Malayalam, and Punjabi. Each of these news datasets comprises 10 or more classes of news articles. L3Cube-IndicNews offers 3 distinct datasets tailored to handle different document lengths that are classified as: Short Headlines Classification (SHC) dataset containing the news headline and news category, Long Document Classification (LDC) dataset containing the whole news article and the news category, and Long Paragraph Classification (LPC) containing sub-articles of the news and the news category. We maintain consistent labeling across all 3 datasets for in-depth length-based analysis. We evaluate each of these Indic language datasets using 4 different models including monolingual BERT, multilingual Indic Sentence BERT (IndicSBERT), and IndicBERT. This research contributes significantly to expanding the pool of available text classification datasets and also makes it possible to develop topic classification models for Indian regional languages. This also serves as an excellent resource for cross-lingual analysis owing to the high overlap of labels among languages. The datasets and models are shared publicly at https://github.com/l3cube-pune/indic-nlp

View on arXivView PDF

Code Implementations(1)

l3cube-pune/indic-nlpOfficial100%

This repository is describes the Indic NLP resources from L3Cube.

231Oct 16, 20231 years ago

Similar Papers

L3Cube-MahaNews: News-based Short Text and Long Document Classification Datasets in Marathi

91% match1 repo1602024

L3Cube-IndicHeadline-ID: A Dataset for Headline Identification and Semantic Evaluation in Low-Resource Indian Languages

87% match1 repo232025

L3Cube-MahaSum: A Comprehensive Dataset and BART Models for Abstractive Text Summarization in Marathi

83% match1 repo1602024

A Breadth-First Catalog of Text Processing, Speech Processing and Multimodal Research in South Asian Languages

83% match2024

L3Cube-IndicQuest: A Benchmark Question Answering Dataset for Evaluating Knowledge of LLMs in Indic Context

83% match1 repo232024

L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT

82% match2023

Cite this paper

@article{mirashi2024lcubeindicnews,
  title  = {L3Cube-IndicNews: News-based Short Text and Long Document Classification Datasets in Indic Languages},
  author = {Aishwarya Mirashi and Srushti Sonavane and Purva Lingayat and Tejas Padhiyar and Raviraj Joshi},
  year   = {2024},
  eprint = {2401.02254},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2401.02254v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2401.02254.svg)](https://codeofpaper.com/paper/2401.02254)

Discussion