How Good is the Model in Model-in-the-loop Event Coreference Resolution Annotation?
Abstract
Annotating cross-document event coreference links is a time-consuming and cognitively demanding task that can compromise annotation quality and efficiency. To address this, we propose a model-in-the-loop annotation approach for event coreference resolution, where a machine learning model suggests likely corefering event pairs only. We evaluate the effectiveness of this approach by first simulating the annotation process and then, using a novel annotator-centric Recall-Annotation effort trade-off metric, we compare the results of various underlying models and datasets. We finally present a method for obtaining 97\% recall while substantially reducing the workload required by a fully manual annotation process. Code and data can be found at https://github.com/ahmeshaf/model_in_coref
Cite
@article{arxiv.2306.05434,
title = {How Good is the Model in Model-in-the-loop Event Coreference Resolution Annotation?},
author = {Shafiuddin Rehan Ahmed and Abhijnan Nath and Michael Regan and Adam Pollins and Nikhil Krishnaswamy and James H. Martin},
journal= {arXiv preprint arXiv:2306.05434},
year = {2023}
}
Comments
The 17th Liguistics Annotation Workshop, 2023 (LAW-XVII) short paper. 10 pages, 6 figures, 1 table