To bridge the gap between performance-oriented benchmarks and the evaluation of cognitively inspired models, we introduce BLiSS 1.0, a Benchmark of Learner Interlingual Syntactic Structure. Our benchmark operationalizes a new paradigm of selective tolerance, testing whether a model finds a naturalistic learner error more plausible than a matched, artificial error within the same sentence. Constructed from over 2.8 million naturalistic learner sentences, BLiSS provides 136,867 controlled triplets (corrected, learner, artificial) for this purpose. Experiments on a diverse suite of models demonstrate that selective tolerance is a distinct capability from standard grammaticality, with performance clustering strongly by training paradigm. This validates BLiSS as a robust tool for measuring how different training objectives impact a model's alignment with the systematic patterns of human language acquisition.
@article{arxiv.2510.19419,
title = {BLiSS 1.0: Evaluating Bilingual Learner Competence in Second Language Small Language Models},
author = {Yuan Gao and Suchir Salhan and Andrew Caines and Paula Buttery and Weiwei Sun},
journal= {arXiv preprint arXiv:2510.19419},
year = {2025}
}
Comments
Accepted Paper at the BabyLM Workshop 2025 @ EMNLP (Presentation in Suzhou, China)