Usability evaluation is an essential method to support the design of effective and intuitive user interfaces (UIs). However, it commonly relies on resource-intensive, expert-driven methods, which limit its accessibility, especially for small organizations. Recent multimodal large language models (MLLMs) have the potential to support usability evaluation by analyzing textual instructions together with visual UI context. This paper investigates the use of MLLMs as assistive tools for usability evaluation by framing the task as a prioritization problem. It identifies and explains usability issues and ranks them by severity. We report a study that compares the evaluations generated by multiple MLLMs with assessments from usability experts. The results demonstrate that MLLMs can offer complementary insights and support the efficient prioritization of critical issues. Additionally, we present an interactive visualization tool that enables the transparent review and validation of model-generated findings. Based on this, we outline concepts for integrating MLLM-based usability evaluation into real-world development workflows.
@article{arxiv.2508.16165,
title = {Investigating Multimodal Large Language Models to Support Usability Evaluation},
author = {Sebastian Lubos and Alexander Felfernig and Damian Garber and Gerhard Leitner and Julian Schwazer and Manuel Henrich},
journal= {arXiv preprint arXiv:2508.16165},
year = {2026}
}
Comments
To appear in the Proceedings of IEA/AIE 2026, Springer LNAI