PuoBERTa: Training and evaluation of a curated language model for Setswana

Vukosi Marivate, Moseli Mots'Oehli, Valencia Wagner, Richard Lastrucci, Isheanesu Dzingirai
10/13/2023
cs.CL

No official code found. Best implementation: dsfsi/PuoBERTa (6 stars, last commit 2025-12-30). 1 more below.

Abstract

Natural language processing (NLP) has made significant progress for well-resourced languages such as English but lagged behind for low-resource languages like Setswana. This paper addresses this gap by presenting PuoBERTa, a customised masked language model trained specifically for Setswana. We cover how we collected, curated, and prepared diverse monolingual texts to generate a high-quality corpus for PuoBERTa's training. Building upon previous efforts in creating monolingual resources for Setswana, we evaluated PuoBERTa across several NLP tasks, including part-of-speech (POS) tagging, named entity recognition (NER), and news categorisation. Additionally, we introduced a new Setswana news categorisation dataset and provided the initial benchmarks using PuoBERTa. Our work demonstrates the efficacy of PuoBERTa in fostering NLP capabilities for understudied languages like Setswana and paves the way for future research directions.

View on arXivView PDF

Code Implementations(2)

dsfsi/PuoBERTaCommunity100%

A Roberta-based language model specially designed for Setswana, using the new PuoData dataset.

62Python, MakefileOct 12, 20239 months agoNOASSERTION
african-languagesafricannlpdsfsi-datasetsnlprocsetswana+2 more
dsfsi/PuoDataCommunity100%
CC-BY-SA-4.0

Curated corpora for Setswana. Used to train PuoBERTa.

41Oct 12, 20232 years agoCC-BY-SA-4.0
african-languagesafrican-nlpcorporadsfsi-datasetsnatural-language-processing+4 more

Similar Papers

Data-Efficient French Language Modeling with CamemBERTa

90% match2023

Conversations in Galician: a Large Language Model for an Underrepresented Language

87% match2023

SwissBERT: The Multilingual Language Model for Switzerland

87% match1 repo292023

Spanish Pre-trained BERT Model and Evaluation Data

87% match2 repos5072023

Language Model Adaptation to Specialized Domains through Selective Masking based on Genre and Topical Characteristics

86% match1 repo42024

MasakhaNEWS: News Topic Classification for African languages

86% match1 repo262023

Cite this paper

@article{marivate2023puoberta,
  title  = {PuoBERTa: Training and evaluation of a curated language model for Setswana},
  author = {Vukosi Marivate and Moseli Mots'Oehli and Valencia Wagner and Richard Lastrucci and Isheanesu Dzingirai},
  year   = {2023},
  eprint = {2310.09141},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.09141v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.09141.svg)](https://codeofpaper.com/paper/2310.09141)

Discussion