English

Synthetic Student Responses: LLM-Extracted Features for IRT Difficulty Parameter Estimation

Computers and Society 2026-02-03 v1 Artificial Intelligence

Abstract

Educational assessment relies heavily on knowing question difficulty, traditionally determined through resource-intensive pre-testing with students. This creates significant barriers for both classroom teachers and assessment developers. We investigate whether Item Response Theory (IRT) difficulty parameters can be accurately estimated without student testing by modeling the response process and explore the relative contribution of different feature types to prediction accuracy. Our approach combines traditional linguistic features with pedagogical insights extracted using Large Language Models (LLMs), including solution step count, cognitive complexity, and potential misconceptions. We implement a two-stage process: first training a neural network to predict how students would respond to questions, then deriving difficulty parameters from these simulated response patterns. Using a dataset of over 250,000 student responses to mathematics questions, our model achieves a Pearson correlation of approximately 0.78 between predicted and actual difficulty parameters on completely unseen questions.

Keywords

Cite

@article{arxiv.2602.00034,
  title  = {Synthetic Student Responses: LLM-Extracted Features for IRT Difficulty Parameter Estimation},
  author = {Matias Hoyl},
  journal= {arXiv preprint arXiv:2602.00034},
  year   = {2026}
}

Comments

17 pages, 7 figures

R2 v1 2026-07-01T09:28:18.612Z