PLaMo-100B: A Ground-Up Language Model Designed for Japanese Proficiency

Preferred Elements, :, Kenshin Abe, Kaizaburo Chubachi, Yasuhiro Fujita +16 more
10/10/2024
cs.CLcs.AIcs.LG

No confident code implementation found for this paper.

Abstract

We introduce PLaMo-100B, a large-scale language model designed for Japanese proficiency. The model was trained from scratch using 2 trillion tokens, with architecture such as QK Normalization and Z-Loss to ensure training stability during the training process. Post-training techniques, including Supervised Fine-Tuning and Direct Preference Optimization, were applied to refine the model's performance. Benchmark evaluations suggest that PLaMo-100B performs well, particularly in Japanese-specific tasks, achieving results that are competitive with frontier models like GPT-4. The base model is available at https://huggingface.co/pfnet/plamo-100b.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

PLaMo 2 Technical Report

90% match2025

Rapidly Developing High-quality Instruction Data and Evaluation Benchmark for Large Language Models with Minimal Human Effort: A Case Study on Japanese

86% match2 repos332024

Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities

83% match2024

Kanana: Compute-efficient Bilingual Language Models

82% match1 repo2802025

Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMs

82% match2024

Typhoon-S: Minimal Open Post-Training for Sovereign Large Language Models

82% match2026

Cite this paper

@article{elements2024plamob,
  title  = {PLaMo-100B: A Ground-Up Language Model Designed for Japanese Proficiency},
  author = {Preferred Elements and : and Kenshin Abe and Kaizaburo Chubachi and Yasuhiro Fujita and Yuta Hirokawa and Kentaro Imajo and Toshiki Kataoka and Hiroyoshi Komatsu and Hiroaki Mikami and Tsuguo Mogami and Shogo Murai and Kosuke Nakago and Daisuke Nishino and Toru Ogawa and Daisuke Okanohara and Yoshihiko Ozaki and Shotaro Sano and Shuji Suzuki and Tianqi Xu and Toshihiko Yanase},
  year   = {2024},
  eprint = {2410.07563},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.07563v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.07563.svg)](https://codeofpaper.com/paper/2410.07563)

Discussion