Toward a Perspectivist Turn in Ground Truthing for Predictive Computing
Abstract
Most Artificial Intelligence applications are based on supervised machine learning (ML), which ultimately grounds on manually annotated data. The annotation process is often performed in terms of a majority vote and this has been proved to be often problematic, as highlighted by recent studies on the evaluation of ML models. In this article we describe and advocate for a different paradigm, which we call data perspectivism, which moves away from traditional gold standard datasets, towards the adoption of methods that integrate the opinions and perspectives of the human subjects involved in the knowledge representation step of ML processes. Drawing on previous works which inspired our proposal we describe the potential of our proposal for not only the more subjective tasks (e.g. those related to human language) but also to tasks commonly understood as objective (e.g. medical decision making), and present the main advantages of adopting a perspectivist stance in ML, as well as possible disadvantages, and various ways in which such a stance can be implemented in practice. Finally, we share a set of recommendations and outline a research agenda to advance the perspectivist stance in ML.
Cite
@article{arxiv.2109.04270,
title = {Toward a Perspectivist Turn in Ground Truthing for Predictive Computing},
author = {Valerio Basile and Federico Cabitza and Andrea Campagner and Michael Fell},
journal= {arXiv preprint arXiv:2109.04270},
year = {2023}
}
Comments
If you wish to cite this work, consider citing the AAAI 2023 proceedings version (https://doi.org/10.1609/aaai.v37i6.25840) and citing it in this way: Cabitza, F., Campagner, A., & Basile, V. (2023). Toward a Perspectivist Turn in Ground Truthing for Predictive Computing. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6), 6860-6868. https://doi.org/10.1609/aaai.v37i6.25840