MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation
Abstract
As generic machine translation (MT) quality has improved, the need for targeted benchmarks that explore fine-grained aspects of quality has increased. In particular, gender accuracy in translation can have implications in terms of output fluency, translation accuracy, and ethics. In this paper, we introduce MT-GenEval, a benchmark for evaluating gender accuracy in translation from English into eight widely-spoken languages. MT-GenEval complements existing benchmarks by providing realistic, gender-balanced, counterfactual data in eight language pairs where the gender of individuals is unambiguous in the input segment, including multi-sentence segments requiring inter-sentential gender agreement. Our data and code is publicly available under a CC BY SA 3.0 license.
Keywords
Cite
@article{arxiv.2211.01355,
title = {MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation},
author = {Anna Currey and Maria Nădejde and Raghavendra Pappagari and Mia Mayer and Stanislas Lauly and Xing Niu and Benjamin Hsu and Georgiana Dinu},
journal= {arXiv preprint arXiv:2211.01355},
year = {2022}
}
Comments
Accepted at EMNLP 2022. Data and code: https://github.com/amazon-research/machine-translation-gender-eval