Mask and You Shall Receive: Optimizing Masked Language Modeling For Pretraining BabyLMs

Lukas Edman, Alexander Fraser
10/23/2025
cs.CL

No confident code implementation found for this paper.

Abstract

We describe our strategy for the 2025 edition of the BabyLM Challenge. Our main contribution is that of an improved form of Masked Language Modeling (MLM), which adapts the probabilities of the tokens masked according to the model's ability to predict them. The results show a substantial increase in performance on (Super)GLUE tasks over the standard MLM. We also incorporate sub-token embeddings, finding that this increases the model's morphological generalization capabilities. Our submission beats the baseline in the strict-small track.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

AntLM: Bridging Causal and Masked Language Models

87% match2024

Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

86% match2024

Masked Diffusion Language Models with Frequency-Informed Training

86% match1 repo162025

Representation Deficiency in Masked Language Modeling

85% match1 repo102023

Should We Still Pretrain Encoders with Masked Language Modeling?

85% match2025

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models

85% match2025

Cite this paper

@article{edman2025mask,
  title  = {Mask and You Shall Receive: Optimizing Masked Language Modeling For Pretraining BabyLMs},
  author = {Lukas Edman and Alexander Fraser},
  year   = {2025},
  eprint = {2510.20475},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2510.20475v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2510.20475.svg)](https://codeofpaper.com/paper/2510.20475)

Discussion