Related papers: Analyzing Force Concept Inventory with Item Respon…
Within the educational context, students' assessment tests are routinely validated through Item Response Theory (IRT) models which assume unidimensionality and absence of Differential Item Functioning (DIF). In this paper, we investigate if…
Multimodal Large Language Models (MLLMs) have recently emerged as general architectures capable of reasoning over diverse modalities. Benchmarks for MLLMs should measure their ability for cross-modal integration. However, current benchmarks…
Automated short answer grading (ASAG) with large language models (LLMs) is commonly evaluated with aggregate metrics such as macro-F1 and Cohen's kappa. However, these metrics provide limited insight into how grading performance varies…
Multidimensional item response theory is a statistical test theory used to estimate the latent skills of learners and the difficulty levels of problems based on test results. Both compensatory and non-compensatory models have been proposed…
Traditional problem-based exams are not efficient instruments for assessing the "structure" of physics students' conceptual knowledge or for providing diagnostically detailed feedback to students and teachers. We present the Free Term Entry…
Generative AI technologies such as large language models show novel potentials to enhance educational research. For example, generative large language models were shown to be capable to solve quantitative reasoning tasks in physics and…
Physics education researchers have developed many evidence-based instructional strategies to enhance physics students' conceptual learning. These strategies have historically been tested using assessments such as the Force Concept Inventory…
We describe the construction, validation and testing of a concept inventory for an Introduction to Physics of Semiconductors course offered by the department of physics for undergraduate engineering students. By design, this inventory…
The proliferation of Large Language Models (LLMs) necessitates valid evaluation methods to guide downstream applications and actionable future improvements. The Item Response Theory (IRT) has recently emerged as a promising framework for…
Statistical models such as those derived from Item Response Theory (IRT) enable the assessment of students on a specific subject, which can be useful for several purposes (e.g., learning path customization, drop-out prediction). However,…
We introduce an Integrative Ranking and Thresholding (IRT) framework for fusing evidence from multiple testing procedures. The key innovation is a method that transforms binary testing decisions into compound $e-$values, enabling the…
We propose a dyadic Item Response Theory (dIRT) model for measuring interactions of pairs of individuals when the responses to items represent the actions (or behaviors, perceptions, etc.) of each individual (actor) made within the context…
Item response theory (IRT) models are widely used in psychometrics and educational measurement, being deployed in many high stakes tests such as the GRE aptitude test. IRT has largely focused on estimation of a single latent trait (e.g.…
In this paper, we apply Item Response Theory, popular in education and political science research, to the analysis of argument persuasiveness in language. We empirically evaluate the model's performance on three datasets, including a novel…
Scene-aware Complementary Item Retrieval (CIR) is a challenging task which requires to generate a set of compatible items across domains. Due to the subjectivity, it is difficult to set up a rigorous standard for both data collection and…
This paper aims to present an online placement test. It is based on the Item Response Theory to provide relevant estimates of learner competences. The proposed test is the entry point of our e-Learning system. It gathers the learner…
We report on progress towards the development of an Action Concept Inventory (ACI), a test that measures student understanding of action principles in introductory mechanics and optics. The ACI also covers key concepts of many-paths quantum…
Recent years have witnessed a surge in the number of large language models (LLMs), yet efficiently managing and utilizing these vast resources remains a significant challenge. In this work, we explore how to learn compact representations of…
Missingness is a common occurrence in educational assessment and psychological measurement. It could not be casually ignored as it may threaten the validity of the test if not handled properly. Considering the difference between omitted and…
Evaluating large language models (LLMs) on comprehensive benchmarks is a cornerstone of their development, yet it's often computationally and financially prohibitive. While Item Response Theory (IRT) offers a promising path toward…