How Well Do LLMs Handle Cantonese? Benchmarking Cantonese Capabilities of Large Language Models

Jiyue Jiang, Pengan Chen, Liheng Chen, Sheng Wang, Qinghang Bao +3 more
8/29/2024
cs.CL

No official code found. Best implementation: jiangjyjy/Yue-Benchmark (13 stars, MIT, last commit 2025-07-02).

Abstract

The rapid evolution of large language models (LLMs) has transformed the competitive landscape in natural language processing (NLP), particularly for English and other data-rich languages. However, underrepresented languages like Cantonese, spoken by over 85 million people, face significant development gaps, which is particularly concerning given the economic significance of the Guangdong-Hong Kong-Macau Greater Bay Area, and in substantial Cantonese-speaking populations in places like Singapore and North America. Despite its wide use, Cantonese has scant representation in NLP research, especially compared to other languages from similarly developed regions. To bridge these gaps, we outline current Cantonese NLP methods and introduce new benchmarks designed to evaluate LLM performance in factual generation, mathematical logic, complex reasoning, and general knowledge in Cantonese, which aim to advance open-source Cantonese LLM technology. We also propose future research directions and recommended models to enhance Cantonese LLM development.

View on arXivView PDF

Code Implementations(1)

jiangjyjy/Yue-BenchmarkCommunity100%
MIT

[NAACL 2025] How Well Do LLMs Handle Cantonese? Benchmarking Cantonese Capabilities of Large Language Models

131Shell, PythonJun 1, 20241 years agoMIT

Similar Papers

Developing and Utilizing a Large-Scale Cantonese Dataset for Multi-Tasking in Large Language Models

93% match2025

\textsc{CantoNLU}: A benchmark for Cantonese natural language understanding

91% match2025

HKCanto-Eval: A Benchmark for Evaluating Cantonese Language Understanding and Cultural Comprehension in LLMs

90% match1 repo212025

Measuring Hong Kong Massive Multi-Task Language Understanding

87% match2025

Evaluating Modern Large Language Models on Low-Resource and Morphologically Rich Languages:A Cross-Lingual Benchmark Across Cantonese, Japanese, and Turkish

85% match2025

CantonMT: Cantonese to English NMT Platform with Fine-Tuned Models Using Synthetic Back-Translation Data

85% match1 repo42024

Cite this paper

@article{jiang2024well,
  title  = {How Well Do LLMs Handle Cantonese? Benchmarking Cantonese Capabilities of Large Language Models},
  author = {Jiyue Jiang and Pengan Chen and Liheng Chen and Sheng Wang and Qinghang Bao and Lingpeng Kong and Yu Li and Chuan Wu},
  year   = {2024},
  eprint = {2408.16756},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2408.16756v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2408.16756.svg)](https://codeofpaper.com/paper/2408.16756)

Discussion