Extending the Pre-Training of BLOOM for Improved Support of Traditional Chinese: Models, Methods and Results

Philipp Ennen, Po-Chun Hsu, Chan-Jan Hsu, Chang-Le Liu, Yen-Chen Wu +4 more
3/8/2023
cs.CLcs.AI

No confident code implementation found for this paper. 2 possible matches can be shown below.

Abstract

In this paper we present the multilingual language model BLOOM-zh that features enhanced support for Traditional Chinese. BLOOM-zh has its origins in the open-source BLOOM models presented by BigScience in 2022. Starting from released models, we extended the pre-training of BLOOM by additional 7.4 billion tokens in Traditional Chinese and English covering a variety of domains such as news articles, books, encyclopedias, educational materials as well as spoken language. In order to show the properties of BLOOM-zh, both existing and newly created benchmark scenarios are used for evaluating the performance. BLOOM-zh outperforms its predecessor on most Traditional Chinese benchmarks while maintaining its English capability. We release all our models to the research community.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. 2 weaker matches are hidden by default — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Models

86% match2023

Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM

86% match1 repo112023

TaCo: Enhancing Cross-Lingual Transfer for Low-Resource Languages in LLMs through Translation-Assisted Chain-of-Thought Processes

85% match1 repo142023

Lost in Translation: Large Language Models in Non-English Content Analysis

85% match2023

JIANG: Chinese Open Foundation Language Model

85% match2023

Multilingual Text Representation

85% match2023

Cite this paper

@article{ennen2023extending,
  title  = {Extending the Pre-Training of BLOOM for Improved Support of Traditional Chinese: Models, Methods and Results},
  author = {Philipp Ennen and Po-Chun Hsu and Chan-Jan Hsu and Chang-Le Liu and Yen-Chen Wu and Yin-Hsiang Liao and Chin-Tung Lin and Da-Shan Shiu and Wei-Yun Ma},
  year   = {2023},
  eprint = {2303.04715},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2303.04715v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2303.04715.svg)](https://codeofpaper.com/paper/2303.04715)

Discussion