English

Shared Task on Evaluating Accuracy in Natural Language Generation

Computation and Language 2020-11-10 v2
Authors: Ehud Reiter , Craig Thomson
View on arXiv ↗ PDF ↗

Abstract

We propose a shared task on methodologies and algorithms for evaluating the accuracy of generated texts. Participants will measure the accuracy of basketball game summaries produced by NLG systems from basketball box score data.

Keywords

natural language generation evaluation text generation evaluation metrics

Cite

@article{arxiv.2006.12234,
  title  = {Shared Task on Evaluating Accuracy in Natural Language Generation},
  author = {Ehud Reiter and Craig Thomson},
  journal= {arXiv preprint arXiv:2006.12234},
  year   = {2020}
}

Comments

To appear in INLG 2020

Related papers

View all related →
Computation and Language · Computer Science
Generation Challenges: Results of the Accuracy Evaluation Shared Task
Craig Thomson, Ehud Reiter
2021-08-17
Computation and Language · Computer Science
A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems
Craig Thomson, Ehud Reiter
2020-11-10
Computation and Language · Computer Science
Evaluation of Text Generation: A Survey
Asli Celikyilmaz, Elizabeth Clark, Jianfeng Gao
2021-05-19
Computation and Language · Computer Science
Human or Machine: Automating Human Likeliness Evaluation of NLG Texts
Erion Çano, Ondřej Bojar
2020-06-08
Computation and Language · Computer Science
Decision-Oriented Text Evaluation
Yu-Shiang Huang, Chuan-Ju Wang, Chung-Chi Chen
2025-07-04
Computation and Language · Computer Science
Grammar Accuracy Evaluation (GAE): Quantifiable Quantitative Evaluation of Machine Translation Models
Dojun Park, Youngjin Jang, Harksoo Kim
2022-08-04
Computation and Language · Computer Science
Automating Text Naturalness Evaluation of NLG Systems
Erion Çano, Ondřej Bojar
2020-06-25
Computation and Language · Computer Science
Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation
Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric P. Xing +1
2022-01-25
Computation and Language · Computer Science
Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation
Cristina Garbacea, Samuel Carton, Shiyan Yan, Qiaozhu Mei
2019-09-09
Computation and Language · Computer Science
Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices
Patrícia Schmidtová, Saad Mahamood, Simone Balloccu, Ondřej Dušek +5
2024-08-20
Computation and Language · Computer Science
Perturbation CheckLists for Evaluating NLG Evaluation Metrics
Ananya B. Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan +1
2021-09-14
Computation and Language · Computer Science
Consistency Evaluation of News Article Summaries Generated by Large (and Small) Language Models
Colleen Gilhuly, Haleh Shahzad
2025-03-03
Computation and Language · Computer Science
A Survey of Natural Language Generation
Chenhe Dong, Yinghui Li, Haifan Gong, Miaoxin Chen +3
2022-08-03
Computation and Language · Computer Science
The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Results
Marina Fomicheva, Piyawat Lertvittayakumjorn, Wei Zhao, Steffen Eger +1
2021-10-12
Computation and Language · Computer Science
Neural Language Generation: Formulation, Methods, and Evaluation
Cristina Garbacea, Qiaozhu Mei
2020-08-03
Computation and Language · Computer Science
On conducting better validation studies of automatic metrics in natural language generation evaluation
Johnny Tian-Zheng Wei
2019-08-01
Computation and Language · Computer Science
BARTScore: Evaluating Generated Text as Text Generation
Weizhe Yuan, Graham Neubig, Pengfei Liu
2021-10-28
Computation and Language · Computer Science
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation
Albert Gatt, Emiel Krahmer
2018-01-30
Computation and Language · Computer Science
What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability
Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernández +1
2023-10-23
Computation and Language · Computer Science
FactSim: Fact-Checking for Opinion Summarization
Leandro Anghinoni, Jorge Sanchez
2026-02-10
Computation and Language · Computer Science
Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methods
Wei Li, Wenhao Wu, Moye Chen, Jiachen Liu +2
2022-03-11
Computation and Language · Computer Science
Towards Pragmatic Production Strategies for Natural Language Generation Tasks
Mario Giulianelli
2022-10-25
Computation and Language · Computer Science
Automatic Quality Estimation for Natural Language Generation: Ranting (Jointly Rating and Ranking)
Ondřej Dušek, Karin Sevegnani, Ioannis Konstas, Verena Rieser
2019-10-11
Computation and Language · Computer Science
Automatic and Human-AI Interactive Text Generation
Yao Dou, Philippe Laban, Claire Gardent, Wei Xu
2023-10-09
Computation and Language · Computer Science
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
Ozan Caglayan, Pranava Madhyastha, Lucia Specia
2020-10-27
R2 v1 2026-06-23T16:31:09.896Z