The development of Artificial Intelligence (AI), including AI in Science (AIS), should be done following the principles of responsible AI. Progress in responsible AI is often quantified through evaluation metrics, yet there has been less work on assessing the robustness and reliability of the metrics themselves. We reflect on prior work that examines the robustness of fairness metrics for recommender systems as a type of AI application and summarise their key takeaways into a set of non-exhaustive guidelines for developing reliable metrics of responsible AI. Our guidelines apply to a broad spectrum of AI applications, including AIS.
@article{arxiv.2510.26007,
title = {The Quest for Reliable Metrics of Responsible AI},
author = {Theresia Veronika Rampisela and Maria Maistro and Tuukka Ruotsalo and Christina Lioma},
journal= {arXiv preprint arXiv:2510.26007},
year = {2025}
}
Comments
Accepted for presentation at the AI in Science Summit 2025