Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation

Zhouhong Gu, Xiaoxuan Zhu, Haoning Ye, Lin Zhang, Jianchen Wang +14 more
6/9/2023
cs.CL

Official code: MikeGu721/XiezhiBenchmark (98 stars, last commit 2023-12-05).

Abstract

New Natural Langauge Process~(NLP) benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present Xiezhi, the most comprehensive evaluation suite designed to assess holistic domain knowledge. Xiezhi comprises multiple-choice questions across 516 diverse disciplines ranging from 13 different subjects with 249,587 questions and accompanied by Xiezhi-Specialty and Xiezhi-Interdiscipline, both with 15k questions. We conduct evaluation of the 47 cutting-edge LLMs on Xiezhi. Results indicate that LLMs exceed average performance of humans in science, engineering, agronomy, medicine, and art, but fall short in economics, jurisprudence, pedagogy, literature, history, and management. We anticipate Xiezhi will help analyze important strengths and shortcomings of LLMs, and the benchmark is released in~\url{https://github.com/MikeGu721/XiezhiBenchmark}.

View on arXivView PDF

Code Implementations(1)

MikeGu721/XiezhiBenchmarkOfficial100%
984Shell, PythonJun 7, 20232 years ago

Similar Papers

ArcMMLU: A Library and Information Science Benchmark for Large Language Models

91% match1 repo162023

C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models

91% match1 repo1.9k2026

BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXams

90% match1 repo202023

M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Models

90% match1 repo1082023

Domain Mastery Benchmark: An Ever-Updating Benchmark for Evaluating Holistic Domain Knowledge of Large Language Model--A Preliminary Release

89% match2023

MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models

89% match2023

Cite this paper

@article{gu2023xiezhi,
  title  = {Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation},
  author = {Zhouhong Gu and Xiaoxuan Zhu and Haoning Ye and Lin Zhang and Jianchen Wang and Yixin Zhu and Sihang Jiang and Zhuozhi Xiong and Zihan Li and Weijie Wu and Qianyu He and Rui Xu and Wenhao Huang and Jingping Liu and Zili Wang and Shusen Wang and Weiguo Zheng and Hongwei Feng and Yanghua Xiao},
  year   = {2023},
  eprint = {2306.05783},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.05783v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.05783.svg)](https://codeofpaper.com/paper/2306.05783)

Discussion