GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
Abstract
Evaluation in machine learning is usually informed by past choices, for example which datasets or metrics to use. This standardization enables the comparison on equal footing using leaderboards, but the evaluation choices become sub-optimal as better alternatives arise. This problem is especially pertinent in natural language generation which requires ever-improving suites of datasets, metrics, and human evaluation to make definitive claims. To make following best model evaluation practices easier, we introduce GEMv2. The new version of the Generation, Evaluation, and Metrics Benchmark introduces a modular infrastructure for dataset, model, and metric developers to benefit from each others work. GEMv2 supports 40 documented datasets in 51 languages. Models for all datasets can be evaluated online and our interactive data card creation and rendering tools make it easier to add new datasets to the living benchmark.
Keywords
Cite
@article{arxiv.2206.11249,
title = {GEMv2: Multilingual NLG Benchmarking in a Single Line of Code},
author = {Sebastian Gehrmann and Abhik Bhattacharjee and Abinaya Mahendiran and Alex Wang and Alexandros Papangelis and Aman Madaan and Angelina McMillan-Major and Anna Shvets and Ashish Upadhyay and Bingsheng Yao and Bryan Wilie and Chandra Bhagavatula and Chaobin You and Craig Thomson and Cristina Garbacea and Dakuo Wang and Daniel Deutsch and Deyi Xiong and Di Jin and Dimitra Gkatzia and Dragomir Radev and Elizabeth Clark and Esin Durmus and Faisal Ladhak and Filip Ginter and Genta Indra Winata and Hendrik Strobelt and Hiroaki Hayashi and Jekaterina Novikova and Jenna Kanerva and Jenny Chim and Jiawei Zhou and Jordan Clive and Joshua Maynez and João Sedoc and Juraj Juraska and Kaustubh Dhole and Khyathi Raghavi Chandu and Laura Perez-Beltrachini and Leonardo F. R. Ribeiro and Lewis Tunstall and Li Zhang and Mahima Pushkarna and Mathias Creutz and Michael White and Mihir Sanjay Kale and Moussa Kamal Eddine and Nico Daheim and Nishant Subramani and Ondrej Dusek and Paul Pu Liang and Pawan Sasanka Ammanamanchi and Qi Zhu and Ratish Puduppully and Reno Kriz and Rifat Shahriyar and Ronald Cardenas and Saad Mahamood and Salomey Osei and Samuel Cahyawijaya and Sanja Štajner and Sebastien Montella and Shailza and Shailza Jolly and Simon Mille and Tahmid Hasan and Tianhao Shen and Tosin Adewumi and Vikas Raunak and Vipul Raheja and Vitaly Nikolaev and Vivian Tsai and Yacine Jernite and Ying Xu and Yisi Sang and Yixin Liu and Yufang Hou},
journal= {arXiv preprint arXiv:2206.11249},
year = {2022}
}