Evaluating UX in the context of AI's complexity, unpredictability, and generative nature presents unique challenges. How can we support HCI researchers to create comprehensive UX evaluation plans? In this paper, we introduce EvAlignUX, a system powered by large language models and grounded in scientific literature, designed to help HCI researchers explore evaluation metrics and their relationship to research outcomes. A user study with 19 HCI scholars showed that EvAlignUX improved the perceived quality and confidence in UX evaluation plans while prompting deeper consideration of research impact and risks. The system enhanced participants' thought processes, leading to the creation of a ``UX Question Bank'' to guide UX evaluation development. Findings also highlight how researchers' backgrounds influence their inspiration and concerns about AI over-reliance, pointing to future research on AI's role in fostering critical thinking. In a world where experience defines impact, we discuss the importance of shifting UX evaluation from a ``method-centric'' to a ``mindset-centric'' approach as the key to meaningful and lasting design evaluation.
@article{arxiv.2409.15471,
title = {EvAlignUX: Advancing UX Evaluation through LLM-Supported Metrics Exploration},
author = {Qingxiao Zheng and Minrui Chen and Pranav Sharma and Yiliu Tang and Mehul Oswal and Yiren Liu and Yun Huang},
journal= {arXiv preprint arXiv:2409.15471},
year = {2025}
}