This study validates Large Language Models (LLMs) as a dynamic alternative to questionnaire-based personality assessment. Using a within-subjects experiment (N=33), we compared Big Five personality scores derived from guided LLM conversations against the gold-standard IPIP-50 questionnaire, while also measuring user-perceived accuracy. Results indicate moderate convergent validity (r=0.38-0.58), with Conscientiousness, Openness, and Neuroticism scores statistically equivalent between methods. Agreeableness and Extraversion showed significant differences, suggesting trait-specific calibration is needed. Notably, participants rated LLM-generated profiles as equally accurate as traditional questionnaire results. These findings suggest conversational AI offers a promising new approach to traditional psychometrics.
@article{arxiv.2602.15848,
title = {Can LLMs Assess Personality? Validating Conversational AI for Trait Profiling},
author = {Andrius Matšenas and Anet Lello and Tõnis Lees and Hans Peep and Kim Lilii Tamm},
journal= {arXiv preprint arXiv:2602.15848},
year = {2026}
}