BERTje: A Dutch BERT Model

Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord +1 more
12/19/2019
cs.CL

Official code: wietsedv/bertje (145 stars, Apache-2.0, last commit 2023-02-16).

Abstract

The transformer-based pre-trained language model BERT has helped to improve state-of-the-art performance on many natural language processing (NLP) tasks. Using the same architecture and parameters, we developed and evaluated a monolingual Dutch BERT model called BERTje. Compared to the multilingual BERT model, which includes Dutch but is only based on Wikipedia text, BERTje is based on a large and diverse dataset of 2.4 billion tokens. BERTje consistently outperforms the equally-sized multilingual BERT model on downstream NLP tasks (part-of-speech tagging, named-entity recognition, semantic role labeling, and sentiment analysis). Our pre-trained Dutch BERT model is made available at https://github.com/wietsedv/bertje.

View on arXivView PDF

Code Implementations(1)

wietsedv/bertjeOfficial100%
Apache-2.0

BERTje is a Dutch pre-trained BERT model developed at the University of Groningen. (EMNLP Findings 2020) "What’s so special about BERT’s layers? A closer look at the NLP pipeline in monolingual and multilingual models"

14510Perl, ShellDec 18, 20193 years agoApache-2.0

Similar Papers

ParsBERT: Transformer-based Model for Persian Language Understanding

89% match1 repo4332020

idT5: Indonesian Version of Multilingual T5 Transformer

88% match2023

PuoBERTa: Training and evaluation of a curated language model for Setswana

88% match2 repos62023

PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing

88% match2023

Data-Efficient French Language Modeling with CamemBERTa

88% match2023

PersianLLaMA: Towards Building First Persian Large Language Model

87% match2023

Cite this paper

@article{vries2019bertje,
  title  = {BERTje: A Dutch BERT Model},
  author = {Wietse de Vries and Andreas van Cranenburgh and Arianna Bisazza and Tommaso Caselli and Gertjan van Noord and Malvina Nissim},
  year   = {2019},
  eprint = {1912.09582},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1912.09582}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1912.09582.svg)](https://codeofpaper.com/paper/1912.09582)

Discussion