Assessing Physics Students' Scientific Argumentation using Natural Language Processing

Scientific argumentation is a core science and engineering practice and a necessary 21st Century workforce skill. Due to the nature of large enrollment classes, it is difficult to individually assess students and provide feedback on their scientific argumentation. The recent developments in Natural Language Processing (NLP) and Machine Learning (ML) provide new opportunities to analyze large collections of student writing efficiently. In this study, we investigate how undergraduate students' scientific argumentation evolves across four semesters of an introductory calculus-based physics course as increasingly structured argumentation scaffolds were introduced. We investigate the use of NLP and ML, specifically topic modeling, to analyze student scientific argumentation across those semesters. We report on the emergent themes present in each semester. Our findings show a clear shift in the thematic focus of student arguments corresponding to the level of scaffolding provided. In semesters with minimal scaffolding, students' arguments emphasized procedural and surface-level features, while semesters with explicit scaffolds exhibited greater concentration around physics-principle-based themes. These results suggest that structured scaffolding supports students in constructing more conceptually grounded scientific arguments and highlights the potential of NLP and ML as scalable approaches for evaluate broad trends in students' scientific argumentation.