Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE

Yixuan Zhang, Haonan Li
10/14/2023
cs.CL

No official code found. Best implementation: isen-zhang/ACLUE (34 stars, MIT, last commit 2024-03-20).

Abstract

Large language models (LLMs) have showcased remarkable capabilities in understanding and generating language. However, their ability in comprehending ancient languages, particularly ancient Chinese, remains largely unexplored. To bridge this gap, we present ACLUE, an evaluation benchmark designed to assess the capability of language models in comprehending ancient Chinese. ACLUE consists of 15 tasks cover a range of skills, spanning phonetic, lexical, syntactic, semantic, inference and knowledge. Through the evaluation of eight state-of-the-art LLMs, we observed a noticeable disparity in their performance between modern Chinese and ancient Chinese. Among the assessed models, ChatGLM2 demonstrates the most remarkable performance, achieving an average score of 37.4%. We have made our code and data public available.

View on arXivView PDF

Code Implementations(1)

isen-zhang/ACLUECommunity100%
MIT

Official github repo for ACLUE, an evaluation benchmark focused on ancient Chinese language comprehension

340Shell, PythonAug 17, 20232 years agoMIT

Similar Papers

HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models

91% match1 repo1382023

Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System

88% match2026

ArcMMLU: A Library and Information Science Benchmark for Large Language Models

88% match1 repo162023

CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks

88% match2026

CMMLU: Measuring massive multitask language understanding in Chinese

87% match1 repo8222023

LLM Probe: Evaluating LLMs for Low-Resource Languages

87% match2026

Cite this paper

@article{zhang2023large,
  title  = {Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE},
  author = {Yixuan Zhang and Haonan Li},
  year   = {2023},
  eprint = {2310.09550},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.09550v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.09550.svg)](https://codeofpaper.com/paper/2310.09550)

Discussion