PhayaThaiBERT: Enhancing a Pretrained Thai Language Model with Unassimilated Loanwords

Panyut Sriwirote, Jalinee Thapiang, Vasan Timtong, Attapol T. Rutherford
11/21/2023
cs.CLcs.AI

No official code found. Best implementation: clicknext-ai/phayathaibert (5 stars, last commit 2023-11-13).

Abstract

While WangchanBERTa has become the de facto standard in transformer-based Thai language modeling, it still has shortcomings in regard to the understanding of foreign words, most notably English words, which are often borrowed without orthographic assimilation into Thai in many contexts. We identify the lack of foreign vocabulary in WangchanBERTa's tokenizer as the main source of these shortcomings. We then expand WangchanBERTa's vocabulary via vocabulary transfer from XLM-R's pretrained tokenizer and pretrain a new model using the expanded tokenizer, starting from WangchanBERTa's checkpoint, on a new dataset that is larger than the one used to train WangchanBERTa. Our results show that our new pretrained model, PhayaThaiBERT, outperforms WangchanBERTa in many downstream tasks and datasets.

View on arXivView PDF

Code Implementations(1)

clicknext-ai/phayathaibertCommunity100%

Source code used in the training of PhayaThaiBERT on a SLURM cluster

52Shell, PythonOct 17, 20232 years ago

Similar Papers

Cabrita: closing the gap for foreign languages

87% match2023

On Significance of Subword tokenization for Low Resource and Efficient Named Entity Recognition: A case study in Marathi

86% match2023

Language Modelling with Pixels

85% match1 repo3482022

Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptation

85% match2023

Increasing The Performance of Cognitively Inspired Data-Efficient Language Models via Implicit Structure Building

85% match2023

ToddlerBERTa: Exploiting BabyBERTa for Grammar Learning and Language Understanding

85% match2023

Cite this paper

@article{sriwirote2023phayathaibert,
  title  = {PhayaThaiBERT: Enhancing a Pretrained Thai Language Model with Unassimilated Loanwords},
  author = {Panyut Sriwirote and Jalinee Thapiang and Vasan Timtong and Attapol T. Rutherford},
  year   = {2023},
  eprint = {2311.12475},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2311.12475v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2311.12475.svg)](https://codeofpaper.com/paper/2311.12475)

Discussion