BERTino: an Italian DistilBERT model

Matteo Muffo, Enrico Bertino
3/31/2023
cs.CLstat.ML

No official code found. Best implementation: indigo-ai/BERTino (3 stars, last commit 2020-10-21).

Abstract

The recent introduction of Transformers language representation models allowed great improvements in many natural language processing (NLP) tasks. However, if on one hand the performances achieved by this kind of architectures are surprising, on the other their usability is limited by the high number of parameters which constitute their network, resulting in high computational and memory demands. In this work we present BERTino, a DistilBERT model which proposes to be the first lightweight alternative to the BERT architecture specific for the Italian language. We evaluated BERTino on the Italian ISDT, Italian ParTUT, Italian WikiNER and multiclass classification tasks, obtaining F1 scores comparable to those obtained by a BERTBASE with a remarkable improvement in training and inference speed.

View on arXivView PDF

Code Implementations(1)

indigo-ai/BERTinoCommunity100%

Repository of BERTino, an Italian DistilBERT model

30Aug 28, 20206 years ago

Similar Papers

Neural Architecture Search for Sentence Classification with BERT

88% match2024

HeRo: RoBERTa and Longformer Hebrew Language Models

86% match2023

MiniRBT: A Two-stage Distilled Small Chinese Pre-trained Model

86% match1 repo3042023

PuoBERTa: Training and evaluation of a curated language model for Setswana

85% match2 repos62023

Data-Efficient French Language Modeling with CamemBERTa

85% match2023

HugNLP: A Unified and Comprehensive Library for Natural Language Processing

85% match1 repo2402023

Cite this paper

@article{muffo2023bertino,
  title  = {BERTino: an Italian DistilBERT model},
  author = {Matteo Muffo and Enrico Bertino},
  year   = {2023},
  eprint = {2303.18121},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2303.18121v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2303.18121.svg)](https://codeofpaper.com/paper/2303.18121)

Discussion