Faithful Chart Summarization with ChaTS-Pi
Abstract
Chart-to-summary generation can help explore data, communicate insights, and help the visually impaired people. Multi-modal generative models have been used to produce fluent summaries, but they can suffer from factual and perceptual errors. In this work we present CHATS-CRITIC, a reference-free chart summarization metric for scoring faithfulness. CHATS-CRITIC is composed of an image-to-text model to recover the table from a chart, and a tabular entailment model applied to score the summary sentence by sentence. We find that CHATS-CRITIC evaluates the summary quality according to human ratings better than reference-based metrics, either learned or n-gram based, and can be further used to fix candidate summaries by removing not supported sentences. We then introduce CHATS-PI, a chart-to-summary pipeline that leverages CHATS-CRITIC during inference to fix and rank sampled candidates from any chart-summarization model. We evaluate CHATS-PI and CHATS-CRITIC using human raters, establishing state-of-the-art results on two popular chart-to-summary datasets.
Cite
@article{arxiv.2405.19094,
title = {Faithful Chart Summarization with ChaTS-Pi},
author = {Syrine Krichene and Francesco Piccinno and Fangyu Liu and Julian Martin Eisenschlos},
journal= {arXiv preprint arXiv:2405.19094},
year = {2024}
}
Comments
To be published in the proceedings of the 2024 Annual Meeting of the Association for Computational Linguistics