As NLP research attracts public attention and excitement, it becomes increasingly important for it to be accessible to a broad audience. As the research community works to democratize NLP, it remains unclear whether beginners to the field can easily apply the latest developments. To understand their needs, we conducted a study with 93 students in an introductory NLP course, where students reproduced results of recent NLP papers. Surprisingly, our results suggest that their technical skill (i.e., programming experience) has limited impact on their effort spent completing the exercise. Instead, we find accessibility efforts by research authors to be key to a successful experience, including thorough documentation and easy access to required models and datasets.
@article{arxiv.2205.02182,
title = {Reproducibility Beyond the Research Community: Experience from NLP Beginners},
author = {Shane Storks and Keunwoo Peter Yu and Joyce Chai},
journal= {arXiv preprint arXiv:2205.02182},
year = {2022}
}
Comments
This version has been removed by arXiv administrators because the submitter did not have the authority to grant the license for this version at the time of submission