English

Towards Neural Language Evaluators

Computation and Language 2019-11-01 v2 Artificial Intelligence Machine Learning
Authors: Hassan Kané , Yusuf Kocyigit , Pelkins Ajanoh , Ali Abdalla , Mohamed Coulibali
View on arXiv ↗ PDF ↗

Abstract

We review three limitations of BLEU and ROUGE -- the most popular metrics used to assess reference summaries against hypothesis summaries, come up with criteria for what a good metric should behave like and propose concrete ways to use recent Transformers-based Language Models to assess reference summaries against hypothesis summaries.

Keywords

natural language generation evaluation evaluation metrics

Cite

@article{arxiv.1909.09268,
  title  = {Towards Neural Language Evaluators},
  author = {Hassan Kané and Yusuf Kocyigit and Pelkins Ajanoh and Ali Abdalla and Mohamed Coulibali},
  journal= {arXiv preprint arXiv:1909.09268},
  year   = {2019}
}

Comments

Accepted to NeurIPS 2019 Document Intelligence Workshop

Related papers

View all related →
Computation and Language · Computer Science
Answers Unite! Unsupervised Metrics for Reinforced Summarization Models
Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano
2019-09-05
Computation and Language · Computer Science
References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluation
Silvia Casola, Yang Janet Liu, Siyao Peng, Oliver Kraus +2
2025-09-17
Computation and Language · Computer Science
Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task
An Yang, Kai Liu, Jing Liu, Yajuan Lyu +1
2018-06-12
Computation and Language · Computer Science
BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust Machine Translation Evaluation
Taisiya Glushkova, Chrysoula Zerva, André F. T. Martins
2023-05-31
Computation and Language · Computer Science
Understanding the Extent to which Summarization Evaluation Metrics Measure the Information Quality of Summaries
Daniel Deutsch, Dan Roth
2020-10-26
Computation and Language · Computer Science
A Critical Study of Automatic Evaluation in Sign Language Translation
Shakib Yazdani, Yasser Hamidullah, Cristina España-Bonet, Eleftherios Avramidis +1
2025-11-17
Computation and Language · Computer Science
A global analysis of metrics used for measuring performance in natural language processing
Kathrin Blagec, Georg Dorffner, Milad Moradi, Simon Ott +1
2022-04-26
Computation and Language · Computer Science
Towards Multiple References Era -- Addressing Data Leakage and Limited Reference Diversity in NLG Evaluation
Xianfeng Zeng, Yijin Liu, Fandong Meng, Jie Zhou
2023-08-11
Computation and Language · Computer Science
Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarization
Itai Mondshine, Tzuf Paz-Argaman, Reut Tsarfaty
2025-07-14
Computation and Language · Computer Science
BLEU might be Guilty but References are not Innocent
Markus Freitag, David Grangier, Isaac Caswell
2020-10-21
Computation and Language · Computer Science
Towards Understanding the Robustness of LLM-based Evaluations under Perturbations
Manav Chaudhary, Harshit Gupta, Savita Bhat, Vasudeva Varma
2024-12-13
Computation and Language · Computer Science
BLEURT: Learning Robust Metrics for Text Generation
Thibault Sellam, Dipanjan Das, Ankur P. Parikh
2020-05-22
Software Engineering · Computer Science
On the Evaluation of Neural Code Summarization
Ensheng Shi, Yanlin Wang, Lun Du, Junjie Chen +4
2022-02-14
Computation and Language · Computer Science
Better Summarization Evaluation with Word Embeddings for ROUGE
Jun-Ping Ng, Viktoria Abrecht
2015-08-26
Computation and Language · Computer Science
A Semantically Motivated Approach to Compute ROUGE Scores
Elaheh ShafieiBavani, Mohammad Ebrahimi, Raymond Wong, Fang Chen
2017-10-23
Human-Computer Interaction · Computer Science
Human-Centered Design Recommendations for LLM-as-a-Judge
Qian Pan, Zahra Ashktorab, Michael Desmond, Martin Santillan Cooper +4
2024-07-08
Computation and Language · Computer Science
Learning by Semantic Similarity Makes Abstractive Summarization Better
Wonjin Yoon, Yoon Sun Yeo, Minbyul Jeong, Bong-Jun Yi +1
2021-06-03
Computation and Language · Computer Science
LLM-ReSum: A Framework for LLM Reflective Summarization through Self-Evaluation
Huyen Nguyen, Haoxuan Zhang, Yang Zhang, Junhua Ding +1
2026-04-29
Computation and Language · Computer Science
A Comparative Study of Quality Evaluation Methods for Text Summarization
Huyen Nguyen, Haihua Chen, Lavanya Pobbathi, Junhua Ding
2024-07-02
Computation and Language · Computer Science
Blind to the Human Touch: Overlap Bias in LLM-Based Summary Evaluation
Jiangnan Fang, Cheng-Tse Liu, Hanieh Deilamsalehy, Nesreen K. Ahmed +4
2026-02-10
Computation and Language · Computer Science
Beyond BLEU: Training Neural Machine Translation with Semantic Similarity
John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, Graham Neubig
2019-09-17
Computation and Language · Computer Science
A Call for Clarity in Reporting BLEU Scores
Matt Post
2018-09-13
Computation and Language · Computer Science
Evaluating Extremely Low-Resource Machine Translation: A Comparative Study of ChrF++ and BLEU Metrics
Sanjeev Kumar, Preethi Jyothi, Pushpak Bhattacharyya
2026-02-20
Computation and Language · Computer Science
Revisiting Summarization Evaluation for Scientific Articles
Arman Cohan, Nazli Goharian
2016-04-05
Computation and Language · Computer Science
Reference-less Quality Estimation of Text Simplification Systems
Louis Martin, Samuel Humeau, Pierre-Emmanuel Mazaré, Antoine Bordes +2
2019-01-31
R2 v1 2026-06-23T11:20:51.531Z