DocAsRef: An Empirical Study on Repurposing Reference-Based Summary Quality Metrics Reference-Freely
Abstract
Automated summary quality assessment falls into two categories: reference-based and reference-free. Reference-based metrics, historically deemed more accurate due to the additional information provided by human-written references, are limited by their reliance on human input. In this paper, we hypothesize that the comparison methodologies used by some reference-based metrics to evaluate a system summary against its corresponding reference can be effectively adapted to assess it against its source document, thereby transforming these metrics into reference-free ones. Experimental results support this hypothesis. After being repurposed reference-freely, the zero-shot BERTScore using the pretrained DeBERTa-large-MNLI model of <0.5B parameters consistently outperforms its original reference-based version across various aspects on the SummEval and Newsroom datasets. It also excels in comparison to most existing reference-free metrics and closely competes with zero-shot summary evaluators based on GPT-3.5.
Keywords
Cite
@article{arxiv.2212.10013,
title = {DocAsRef: An Empirical Study on Repurposing Reference-Based Summary Quality Metrics Reference-Freely},
author = {Forrest Sheng Bao and Ruixuan Tu and Ge Luo and Yinfei Yang and Hebi Li and Minghui Qiu and Youbiao He and Cen Chen},
journal= {arXiv preprint arXiv:2212.10013},
year = {2023}
}
Comments
Accepted into Findings of EMNLP 2023