English

Prompted Zero-Shot Multi-label Classification of Factual Incorrectness in Machine-Generated Summaries

Computation and Language 2023-12-05 v1 Artificial Intelligence

Abstract

This study addresses the critical issue of factual inaccuracies in machine-generated text summaries, an increasingly prevalent issue in information dissemination. Recognizing the potential of such errors to compromise information reliability, we investigate the nature of factual inconsistencies across machine-summarized content. We introduce a prompt-based classification system that categorizes errors into four distinct types: misrepresentation, inaccurate quantities or measurements, false attribution, and fabrication. The participants are tasked with evaluating a corpus of machine-generated summaries against their original articles. Our methodology employs qualitative judgements to identify the occurrence of factual distortions. The results show that our prompt-based approaches are able to detect the type of errors in the summaries to some extent, although there is scope for improvement in our classification systems.

Keywords

Cite

@article{arxiv.2312.01087,
  title  = {Prompted Zero-Shot Multi-label Classification of Factual Incorrectness in Machine-Generated Summaries},
  author = {Aniket Deroy and Subhankar Maity and Saptarshi Ghosh},
  journal= {arXiv preprint arXiv:2312.01087},
  year   = {2023}
}
R2 v1 2026-06-28T13:39:06.801Z