Automatic Error Analysis for Document-level Information Extraction
Abstract
Document-level information extraction (IE) tasks have recently begun to be revisited in earnest using the end-to-end neural network techniques that have been successful on their sentence-level IE counterparts. Evaluation of the approaches, however, has been limited in a number of dimensions. In particular, the precision/recall/F1 scores typically reported provide few insights on the range of errors the models make. We build on the work of Kummerfeld and Klein (2013) to propose a transformation-based framework for automating error analysis in document-level event and (N-ary) relation extraction. We employ our framework to compare two state-of-the-art document-level template-filling approaches on datasets from three domains; and then, to gauge progress in IE since its inception 30 years ago, vs. four systems from the MUC-4 (1992) evaluation.
Cite
@article{arxiv.2209.07442,
title = {Automatic Error Analysis for Document-level Information Extraction},
author = {Aliva Das and Xinya Du and Barry Wang and Kejian Shi and Jiayuan Gu and Thomas Porter and Claire Cardie},
journal= {arXiv preprint arXiv:2209.07442},
year = {2022}
}
Comments
Accepted to ACL 2022 Main Conference. First three authors contributed equally to this work