Explainable Interfaces for Rapid Gaze-Based Interactions in Mixed Reality
Abstract
Gaze-based interactions offer a potential way for users to naturally engage with mixed reality (XR) interfaces. Black-box machine learning models enabled higher accuracy for gaze-based interactions. However, due to the black-box nature of the model, users might not be able to understand and effectively adapt their gaze behaviour to achieve high quality interaction. We posit that explainable AI (XAI) techniques can facilitate understanding of and interaction with gaze-based model-driven system in XR. To study this, we built a real-time, multi-level XAI interface for gaze-based interaction using a deep learning model, and evaluated it during a visual search task in XR. A between-subjects study revealed that participants who interacted with XAI made more accurate selections compared to those who did not use the XAI system (i.e., F1 score increase of 10.8%). Additionally, participants who used the XAI system adapted their gaze behavior over time to make more effective selections. These findings suggest that XAI can potentially be used to assist users in more effective collaboration with model-driven interactions in XR.
Cite
@article{arxiv.2404.13777,
title = {Explainable Interfaces for Rapid Gaze-Based Interactions in Mixed Reality},
author = {Mengjie Yu and Dustin Harris and Ian Jones and Ting Zhang and Yue Liu and Naveen Sendhilnathan and Narine Kokhlikyan and Fulton Wang and Co Tran and Jordan L. Livingston and Krista E. Taylor and Zhenhong Hu and Mary A. Hood and Hrvoje Benko and Tanya R. Jonker},
journal= {arXiv preprint arXiv:2404.13777},
year = {2024}
}