Artificial Intelligence (AI) and large language models (LLMs) are increasingly used in social and psychological research. Among potential applications, LLMs can be used to generate, customise, or adapt measurement instruments. This study presents a preliminary investigation of AI-generated questionnaires by comparing two ChatGPT-based adaptations of the Body Awareness Questionnaire (BAQ) with the validated human-developed version. The AI instruments were designed with different levels of explicitness in content and instructions on construct facets, and their psychometric properties were assessed using a Bayesian Graded Response Model. Results show that although surface wording between AI and original items was similar, differences emerged in dimensionality and in the distribution of item and test information across latent traits. These findings illustrate the importance of applying statistical measures of accuracy to ensure the validity and interpretability of AI-driven tools.
@article{arxiv.2510.24739,
title = {Human- vs. AI-generated tests: dimensionality and information accuracy in latent trait evaluation},
author = {Mario Angelelli and Morena Oliva and Serena Arima and Enrico Ciavolino},
journal= {arXiv preprint arXiv:2510.24739},
year = {2026}
}
Comments
28 pages, 12 figures. Minor corrections and comments added. The published version of this preprint is available in "Statistics" at the following DOI: 10.1080/02331888.2025.2610647