PhoGPT: Generative Pre-training for Vietnamese

Dat Quoc Nguyen, Linh The Nguyen, Chi Tran, Dung Ngoc Nguyen, Dinh Phung +1 more
11/6/2023
cs.CL

Official code: VinAIResearch/PhoGPT (794 stars, BSD-3-Clause, last commit 2024-11-12).

Abstract

We open-source a state-of-the-art 4B-parameter generative model series for Vietnamese, which includes the base pre-trained monolingual model PhoGPT-4B and its chat variant, PhoGPT-4B-Chat. The base model, PhoGPT-4B, with exactly 3.7B parameters, is pre-trained from scratch on a Vietnamese corpus of 102B tokens, with an 8192 context length, employing a vocabulary of 20480 token types. The chat variant, PhoGPT-4B-Chat, is the modeling output obtained by fine-tuning PhoGPT-4B on a dataset of 70K instructional prompts and their responses, along with an additional 290K conversations. In addition, we also demonstrate its superior performance compared to previous open-source models. Our PhoGPT models are available at: https://github.com/VinAIResearch/PhoGPT

View on arXivView PDF

Code Implementations(1)

VinAIResearch/PhoGPTOfficial100%
BSD-3-Clause

PhoGPT: Generative Pre-training for Vietnamese (2023)

79475PythonNov 6, 20231 years agoBSD-3-Clause
generative-pre-trained-transformergptinstruction-followingllmphogpt+2 more

Similar Papers

Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluation

88% match1 repo8.3k2023

Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models

87% match2023

YAYI 2: Multilingual Open-Source Large Language Models

86% match2023

DataComp-LM: In search of the next generation of training sets for language models

86% match2026

Enhancing Chat Language Models by Scaling High-quality Instructional Conversations

86% match1 repo2.9k2023

ASR-Generated Text for Language Model Pre-training Applied to Speech Tasks

85% match2022

Cite this paper

@article{nguyen2023phogpt,
  title  = {PhoGPT: Generative Pre-training for Vietnamese},
  author = {Dat Quoc Nguyen and Linh The Nguyen and Chi Tran and Dung Ngoc Nguyen and Dinh Phung and Hung Bui},
  year   = {2023},
  eprint = {2311.02945},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.02945v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.02945.svg)](https://codeofpaper.com/paper/2311.02945)

Discussion