An Empirical Study of Instruction-tuning Large Language Models in Chinese

Qingyi Si, Tong Wang, Zheng Lin, Xu Zhang, Yanan Cao +1 more
10/11/2023
cs.CLcs.AI

Official code: PhoebusSi/Alpaca-CoT (2,791 stars, Apache-2.0, PyTorch, last commit 2023-12-12).

Abstract

The success of ChatGPT validates the potential of large language models (LLMs) in artificial general intelligence (AGI). Subsequently, the release of LLMs has sparked the open-source community's interest in instruction-tuning, which is deemed to accelerate ChatGPT's replication process. However, research on instruction-tuning LLMs in Chinese, the world's most spoken language, is still in its early stages. Therefore, this paper makes an in-depth empirical study of instruction-tuning LLMs in Chinese, which can serve as a cookbook that provides valuable findings for effectively customizing LLMs that can better respond to Chinese instructions. Specifically, we systematically explore the impact of LLM bases, parameter-efficient methods, instruction data types, which are the three most important elements for instruction-tuning. Besides, we also conduct experiment to study the impact of other factors, e.g., chain-of-thought data and human-value alignment. We hope that this empirical study can make a modest contribution to the open Chinese version of ChatGPT. This paper will release a powerful Chinese LLMs that is comparable to ChatGLM. The code and data are available at https://github.com/PhoebusSi/Alpaca-CoT.

View on arXivView PDF

Code Implementations(1)

PhoebusSi/Alpaca-CoTOfficial100%
PyTorchApache-2.0

We unified the interfaces of instruction-tuning data (e.g., CoT data), multiple LLMs and parameter-efficient methods (e.g., lora, p-tuning) together for easy use. We welcome open-source enthusiasts to initiate any meaningful PR on this repo and integrate as many LLM related technologies as possible. 我们打造了方便研究人员上手和使用大模型等微调平台，我们欢迎开源爱好者发起任何有意义的pr！

2,791248MDX, ShellMar 24, 20232 years agoApache-2.0
alpacachatglmchatgptcotinstruction-tuning+10 more

Similar Papers

Dynamics of Instruction Fine-Tuning for Chinese Large Language Models

91% match1 repo182023

MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuning

91% match2025

From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models

91% match1 repo72023

Demystifying Instruction Mixing for Fine-tuning Large Language Models

90% match1 repo42023

Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases

90% match1 repo8.3k2023

Instruction Tuning for Large Language Models: A Survey

90% match1 repo2322023

Cite this paper

@article{si2023empirical,
  title  = {An Empirical Study of Instruction-tuning Large Language Models in Chinese},
  author = {Qingyi Si and Tong Wang and Zheng Lin and Xu Zhang and Yanan Cao and Weiping Wang},
  year   = {2023},
  eprint = {2310.07328},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.07328v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.07328.svg)](https://codeofpaper.com/paper/2310.07328)

Discussion