Most Natural Language Generation systems need to produce accurate texts. We propose a methodology for high-quality human evaluation of the accuracy of generated texts, which is intended to serve as a gold-standard for accuracy evaluations of data-to-text systems. We use our methodology to evaluate the accuracy of computer generated basketball summaries. We then show how our gold standard evaluation can be used to validate automated metrics
@article{arxiv.2011.03992,
title = {A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems},
author = {Craig Thomson and Ehud Reiter},
journal= {arXiv preprint arXiv:2011.03992},
year = {2020}
}
Comments
To appear in INLG-2020. Resources available at https://github.com/nlgcat/evaluating_accuracy