We introduce open-sci-ref, a family of dense transformer models trained as research baselines across multiple model (0.13B to 1.7B parameters) and token scales (up to 1T) on 8 recent open reference datasets. Evaluating the models on various standardized benchmarks, our training runs set establishes reference points that enable researchers to assess the sanity and quality of alternative training approaches across scales and datasets. Intermediate checkpoints allow comparison and studying of the training dynamics. The established reference baselines allow training procedures to be compared through their scaling trends, aligning them on a common compute axis. Comparison of open reference datasets reveals that training on NemoTron-CC HQ consistently outperforms other reference datasets, followed by DCLM-baseline and FineWeb-Edu. In addition to intermediate training checkpoints, the release includes logs, code, and downstream evaluations to simplify reproduction, standardize comparison, and facilitate future research.
Cite
@article{arxiv.2509.09009,
title = {Open-sci-ref-0.01: open and reproducible reference baselines for language model and dataset comparison},
author = {Marianna Nezhurina and Jörg Franke and Taishi Nakamura and Timur Carstensen and Niccolò Ajroldi and Ville Komulainen and David Salinas and Jenia Jitsev},
journal= {arXiv preprint arXiv:2509.09009},
year = {2026}
}
Comments
v.1.1. AAAI Workshop on Reproducible Artificial Intelligence (RAI, https://reproducibleai.github.io) 2026, camera ready version. Model weights and intermediate training checkpoints are available at https://huggingface.co/collections/open-sci/open-sci-ref-001; code for reproducing training, evaluation and raw experiments data at https://github.com/LAION-AI/open-sci-ref-0.01