It is unclear if existing interpretations of deep neural network models respond effectively to the needs of users. This paper summarizes the common forms of explanations (such as feature attribution, decision rules, or probes) used in over 200 recent papers about natural language processing (NLP), and compares them against user questions collected in the XAI Question Bank. We found that although users are interested in explanations for the road not taken -- namely, why the model chose one result and not a well-defined, seemly similar legitimate counterpart -- most model interpretations cannot answer these questions.
@article{arxiv.2103.14973,
title = {Explaining the Road Not Taken},
author = {Hua Shen and Ting-Hao 'Kenneth' Huang},
journal= {arXiv preprint arXiv:2103.14973},
year = {2021}
}
Comments
Accepted by The 2021 ACM CHI Workshop on Operationalizing Human-Centered Perspectives in Explainable AI (CHI 2021 HCXAI Workshop). For associated website, see https://human-centered-exnlp.github.io