We present a masked diffusion language modeling framework for data-efficient training for the BabyLM 2025 Challenge. Our approach applies diffusion training objectives to language modeling under strict data constraints, incorporating frequency-informed masking that prioritizes learning from rare tokens while maintaining theoretical validity. We explore multiple noise scheduling strategies, including two-mode approaches, and investigate different noise weighting schemes within the NELBO objective. We evaluate our method on the BabyLM benchmark suite, measuring linguistic competence, world knowledge, and human-likeness. Results show performance competitive to hybrid autoregressive-masked baselines, demonstrating that diffusion-based training offers a viable alternative for data-restricted language learning.
@article{arxiv.2509.05056,
title = {Masked Diffusion Language Models with Frequency-Informed Training},
author = {Despoina Kosmopoulou and Efthymios Georgiou and Vaggelis Dorovatas and Georgios Paraskevopoulos and Alexandros Potamianos},
journal= {arXiv preprint arXiv:2509.05056},
year = {2025}
}