ViLexNorm: A Lexical Normalization Corpus for Vietnamese Social Media Text

Thanh-Nhi Nguyen, Thanh-Phong Le, Kiet Van Nguyen
1/29/2024
cs.CL

No official code found. Best implementation: ngxtnhi/ViLexNorm (20 stars, last commit 2024-03-20).

Abstract

Lexical normalization, a fundamental task in Natural Language Processing (NLP), involves the transformation of words into their canonical forms. This process has been proven to benefit various downstream NLP tasks greatly. In this work, we introduce Vietnamese Lexical Normalization (ViLexNorm), the first-ever corpus developed for the Vietnamese lexical normalization task. The corpus comprises over 10,000 pairs of sentences meticulously annotated by human annotators, sourced from public comments on Vietnam's most popular social media platforms. Various methods were used to evaluate our corpus, and the best-performing system achieved a result of 57.74% using the Error Reduction Rate (ERR) metric (van der Goot, 2019a) with the Leave-As-Is (LAI) baseline. For extrinsic evaluation, employing the model trained on ViLexNorm demonstrates the positive impact of the Vietnamese lexical normalization task on other NLP tasks. Our corpus is publicly available exclusively for research purposes.

View on arXivView PDF

Code Implementations(1)

ngxtnhi/ViLexNormCommunity100%

A Lexical Normalization Corpus for Vietnamese Social Media Text

201Jan 30, 20242 years ago
corpusdatasetnlpnormalizationvietnamese-nlp

Similar Papers

ViSoLex: An Open-Source Repository for Vietnamese Social Media Lexical Normalization

90% match1 repo82025

A Weakly Supervised Data Labeling Framework for Machine Lexical Normalization in Vietnamese Social Media

90% match2024

VietNormalizer: An Open-Source, Dependency-Free Python Library for Vietnamese Text Normalization in TTS and NLP Applications

86% match1 repo982026

MultiLexNorm++: A Unified Benchmark and a Generative Model for Lexical Normalization for Asian Languages

83% match2026

ViGoEmotions: A Benchmark Dataset For Fine-grained Emotion Detection on Vietnamese Texts

82% match2026

VlogQA: Task, Dataset, and Baseline Models for Vietnamese Spoken-Based Machine Reading Comprehension

80% match1 repo12024

Cite this paper

@article{nguyen2024vilexnorm,
  title  = {ViLexNorm: A Lexical Normalization Corpus for Vietnamese Social Media Text},
  author = {Thanh-Nhi Nguyen and Thanh-Phong Le and Kiet Van Nguyen},
  year   = {2024},
  eprint = {2401.16403},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2401.16403v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2401.16403.svg)](https://codeofpaper.com/paper/2401.16403)

Discussion