BamiBERT: A New BERT-based Language Model for Vietnamese

Dat Quoc Nguyen, Thinh Pham, Chi Tran, Linh The Nguyen
7/2/2026
cs.CL

No confident code implementation found for this paper.

Abstract

In this paper, we introduce BamiBERT, a new BERT-based pre-trained language model for Vietnamese that addresses key limitations of PhoBERT -- the current de facto Vietnamese text encoder. Trained from scratch on a 129GB corpus of general-domain Vietnamese text for 20 epochs, BamiBERT supports an extended context length of up to 2048 tokens and operates directly on raw input, eliminating the need for external word segmentation. Across 8 Vietnamese benchmarks, it achieves the best score on 11 of 15 metrics and the second-best on 3 others, setting a new state of the art among "base"-sized Vietnamese encoders and demonstrating strong cross-domain generalization. We release BamiBERT at: https://huggingface.co/Qualcomm-AI-Research/BamiBERT

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM-to-MLM Curriculum

87% match2026

Learn Your Tokens: Word-Pooled Tokenization for Language Modeling

86% match1 repo72023

Advancing Neural Encoding of Portuguese with Transformer Albertina PT-*

86% match2023

Learning from Child-Directed Speech in Two-Language Scenarios: A French-English Case Study

85% match2026

IHUBERT: Vector-Based Semantic Deduplication and Domain-Balanced Pretraining for Persian Resources

85% match2026

Data-Efficient French Language Modeling with CamemBERTa

85% match2023

Cite this paper

@article{nguyen2026bamibert,
  title  = {BamiBERT: A New BERT-based Language Model for Vietnamese},
  author = {Dat Quoc Nguyen and Thinh Pham and Chi Tran and Linh The Nguyen},
  year   = {2026},
  eprint = {2607.02259},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2607.02259v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2607.02259.svg)](https://codeofpaper.com/paper/2607.02259)

Discussion