SwissBERT: The Multilingual Language Model for Switzerland

Jannis Vamvas, Johannes Graën, Rico Sennrich
3/23/2023
cs.CL

Official code: ZurichNLP/swissbert (29 stars, MIT, last commit 2024-01-19).

Abstract

We present SwissBERT, a masked language model created specifically for processing Switzerland-related text. SwissBERT is a pre-trained model that we adapted to news articles written in the national languages of Switzerland -- German, French, Italian, and Romansh. We evaluate SwissBERT on natural language understanding tasks related to Switzerland and find that it tends to outperform previous models on these tasks, especially when processing contemporary news and/or Romansh Grischun. Since SwissBERT uses language adapters, it may be extended to Swiss German dialects in future work. The model and our open-source code are publicly released at https://github.com/ZurichNLP/swissbert.

View on arXivView PDF

Code Implementations(1)

ZurichNLP/swissbertOfficial100%
MIT

The multilingual language model for Switzerland

295Shell, PythonMar 20, 20232 years agoMIT

Similar Papers

PuoBERTa: Training and evaluation of a curated language model for Setswana

87% match2 repos62023

Investigating Masking-based Data Generation in Language Models

85% match2023

Vārta: A Large-Scale Headline-Generation Dataset for Indic Languages

84% match1 repo82023

Multilinguals at SemEval-2022 Task 11: Complex NER in Semantically Ambiguous Settings for Low Resource Languages

84% match2022

Language Model Adaptation to Specialized Domains through Selective Masking based on Genre and Topical Characteristics

83% match1 repo42024

Open, Closed, or Small Language Models for Text Classification?

83% match2023

Cite this paper

@article{vamvas2023swissbert,
  title  = {SwissBERT: The Multilingual Language Model for Switzerland},
  author = {Jannis Vamvas and Johannes Graën and Rico Sennrich},
  year   = {2023},
  eprint = {2303.13310},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2303.13310v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2303.13310.svg)](https://codeofpaper.com/paper/2303.13310)

Discussion