DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains

Yanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier, Emmanuel Morin +2 more
4/3/2023
cs.CL

No official code found. Best implementation: qanastek/DrBERT (22 stars, MIT, last commit 2024-02-07).

Abstract

In recent years, pre-trained language models (PLMs) achieve the best performance on a wide range of natural language processing (NLP) tasks. While the first models were trained on general domain data, specialized ones have emerged to more effectively treat specific domains. In this paper, we propose an original study of PLMs in the medical domain on French language. We compare, for the first time, the performance of PLMs trained on both public data from the web and private data from healthcare establishments. We also evaluate different learning strategies on a set of biomedical tasks. In particular, we show that we can take advantage of already existing biomedical PLMs in a foreign language by further pre-train it on our targeted data. Finally, we release the first specialized PLMs for the biomedical field in French, called DrBERT, as well as the largest corpus of medical data under free license on which these models are trained.

View on arXivView PDF

Code Implementations(1)

qanastek/DrBERTCommunity78%
MIT

DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains

223Shell, PythonJan 5, 20232 years agoMIT
bertbiomedicalfrenchlearningmachine+7 more

Similar Papers

BIOptimus: Pre-training an Optimal Biomedical Language Model with Curriculum Learning for Named Entity Recognition

90% match1 repo32023

Harnessing the Power of BERT in the Turkish Clinical Domain: Pretraining Approaches for Limited Data Scenarios

90% match2023

Comprehensive Study on German Language Models for Clinical and Biomedical Text Understanding

90% match2024

EriBERTa: A Bilingual Pre-Trained Language Model for Clinical Natural Language Processing

89% match2023

TCM-GPT: Efficient Pre-training of Large Language Models for Domain Adaptation in Traditional Chinese Medicine

89% match2023

Is Biomedical Specialization Still Worth It? Insights from Domain-Adaptive Language Modelling with a New French Health Corpus

89% match2026

Cite this paper

@article{labrak2023drbert,
  title  = {DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains},
  author = {Yanis Labrak and Adrien Bazoge and Richard Dufour and Mickael Rouvier and Emmanuel Morin and Béatrice Daille and Pierre-Antoine Gourraud},
  year   = {2023},
  eprint = {2304.00958},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.00958v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.00958.svg)](https://codeofpaper.com/paper/2304.00958)

Discussion