Masked Diffusion Language Models with Frequency-Informed Training

Despoina Kosmopoulou, Efthymios Georgiou, Vaggelis Dorovatas, Georgios Paraskevopoulos, Alexandros Potamianos
9/5/2025
cs.CL

No official code found. Best implementation: DespoinaKK/babylm-diffusion (16 stars, MIT, last commit 2025-12-17).

Abstract

We present a masked diffusion language modeling framework for data-efficient training for the BabyLM 2025 Challenge. Our approach applies diffusion training objectives to language modeling under strict data constraints, incorporating frequency-informed masking that prioritizes learning from rare tokens while maintaining theoretical validity. We explore multiple noise scheduling strategies, including two-mode approaches, and investigate different noise weighting schemes within the NELBO objective. We evaluate our method on the BabyLM benchmark suite, measuring linguistic competence, world knowledge, and human-likeness. Results show performance competitive to hybrid autoregressive-masked baselines, demonstrating that diffusion-based training offers a viable alternative for data-restricted language learning.

View on arXivView PDF

Code Implementations(1)

DespoinaKK/babylm-diffusionCommunity64%
MIT

[BabyLM@EMNLP 2025 - Challenge Award] Official Implementation: Masked Diffusion Language Models with Frequency-Informed Training

160Shell, PythonOct 5, 202510 months agoMIT

Similar Papers

Soft-Masked Diffusion Language Models

89% match2025

Simple and Effective Masked Diffusion Language Models

89% match1 repo6992024

What Makes Diffusion Language Models Super Data Learners?

87% match1 repo32025

Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning

87% match1 repo842023

Improved Large Language Diffusion Models

87% match1 repo3.9k2026

Likelihood-Based Diffusion Language Models

87% match1 repo1152023

Cite this paper

@article{kosmopoulou2025masked,
  title  = {Masked Diffusion Language Models with Frequency-Informed Training},
  author = {Despoina Kosmopoulou and Efthymios Georgiou and Vaggelis Dorovatas and Georgios Paraskevopoulos and Alexandros Potamianos},
  year   = {2025},
  eprint = {2509.05056},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2509.05056v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2509.05056.svg)](https://codeofpaper.com/paper/2509.05056)

Discussion