Humans and AIs are often paired on decision tasks with the expectation of achieving complementary performance -- where the combination of human and AI outperforms either one alone. However, how to improve performance of a human-AI team is often not clear without knowing more about what particular information and strategies each agent employs. In this paper, we propose a model based in statistical decision theory to analyze human-AI collaboration from the perspective of what information could be used to improve a human or AI decision. We demonstrate our model on a deepfake detection task to investigate seven video-level features by their unexploited value of information. We compare the human alone, AI alone and human-AI team and offer insights on how the AI assistance impacts people's usage of the information and what information that the AI exploits well might be useful for improving human decisions.
@article{arxiv.2411.10463,
title = {Unexploited Information Value in Human-AI Collaboration},
author = {Ziyang Guo and Yifan Wu and Jason Hartline and Jessica Hullman},
journal= {arXiv preprint arXiv:2411.10463},
year = {2025}
}
Comments
We withdraw this version and direct readers to the updated version available on arXiv. See the new version at arXiv:2502.06152