Inter-Annotator Agreement (IAA) is commonly used as a measure of label consistency in natural language processing tasks. However, in real-world scenarios, IAA has various roles and implications beyond its traditional usage. In this paper, we not only consider IAA as a measure of consistency but also as a versatile tool that can be effectively utilized in practical applications. Moreover, we discuss various considerations and potential concerns when applying IAA and suggest strategies for effectively navigating these challenges.
@article{arxiv.2306.14373,
title = {Inter-Annotator Agreement in the Wild: Uncovering Its Emerging Roles and Considerations in Real-World Scenarios},
author = {NamHyeok Kim and Chanjun Park},
journal= {arXiv preprint arXiv:2306.14373},
year = {2023}
}
Comments
Accepted for Data-centric Machine Learning Research (DMLR) Workshop at ICML 2023