GPT or BERT: why not both?

Lucas Georges Gabriel Charpentier, David Samuel
10/31/2024
cs.CL

No official code found. Best implementation: ltgoslo/gpt-bert (64 stars, MIT, last commit 2025-07-28).

Abstract

We present a simple way to merge masked language modeling with causal language modeling. This hybrid training objective results in a model that combines the strengths of both modeling paradigms within a single transformer stack: GPT-BERT can be transparently used like any standard causal or masked language model. We test the pretraining process that enables this flexible behavior on the BabyLM Challenge 2024. The results show that the hybrid pretraining outperforms masked-only or causal-only models. We openly release the models, training corpora and code.

View on arXivView PDF

Code Implementations(1)

ltgoslo/gpt-bertCommunity100%
MIT

Official implementation of "GPT or BERT: why not both?"

649Shell, PythonOct 28, 20241 years agoMIT

Similar Papers

BERTs are Generative In-Context Learners

87% match1 repo322026

AntLM: Bridging Causal and Masked Language Models

84% match2024

A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT

84% match2023

BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs

84% match2026

BPDec: Unveiling the Potential of Masked Language Modeling Decoder in BERT pretraining

83% match2024

It's All in The [MASK]: Simple Instruction-Tuning Enables BERT-like Masked Language Models As Generative Classifiers

83% match2025

Cite this paper

@article{charpentier2024bert,
  title  = {GPT or BERT: why not both?},
  author = {Lucas Georges Gabriel Charpentier and David Samuel},
  year   = {2024},
  eprint = {2410.24159},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.24159v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.24159.svg)](https://codeofpaper.com/paper/2410.24159)

Discussion