Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability
Applications
2021-06-15 v1 Artificial Intelligence
Social and Information Networks
Abstract
We present a new approach to interpreting IRR that is empirical and contextualized. It is based upon benchmarking IRR against baseline measures in a replication, one of which is a novel cross-replication reliability (xRR) measure based on Cohen's kappa. We call this approach the xRR framework. We opensource a replication dataset of 4 million human judgements of facial expressions and analyze it with the proposed framework. We argue this framework can be used to measure the quality of crowdsourced datasets.
Cite
@article{arxiv.2106.07393,
title = {Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability},
author = {Ka Wong and Praveen Paritosh and Lora Aroyo},
journal= {arXiv preprint arXiv:2106.07393},
year = {2021}
}