RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi +5 more
7/26/2019
cs.CL

Official code: facebookresearch/fairseq (32,213 stars, MIT, PyTorch, last commit 2025-09-30).

Abstract

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

View on arXivView PDF

Code Implementations(1)

facebookresearch/fairseqOfficial100%
PyTorchMIT

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

32,2136,667C++, LuaAug 29, 20171 years agoMIT
artificial-intelligencepythonpytorch

Similar Papers

Fantastic Pretraining Optimizers and Where to Find Them

88% match2025

Test-Time Scaling Makes Overtraining Compute-Optimal

86% match2026

The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysis

85% match2024

Optimal Splitting of Language Models from Mixtures to Specialized Domains

85% match2026

LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty

84% match1 repo232026

Scaling (Down) CLIP: A Comprehensive Analysis of Data, Architecture, and Training Strategies

84% match2024

Cite this paper

@article{liu2019roberta,
  title  = {RoBERTa: A Robustly Optimized BERT Pretraining Approach},
  author = {Yinhan Liu and Myle Ott and Naman Goyal and Jingfei Du and Mandar Joshi and Danqi Chen and Omer Levy and Mike Lewis and Luke Zettlemoyer and Veselin Stoyanov},
  year   = {2019},
  eprint = {1907.11692},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/1907.11692}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/1907.11692.svg)](https://codeofpaper.com/paper/1907.11692)

Discussion