Owing to the lack of corpora for low-resource languages, current works on dialogue generation have mainly focused on English. In this paper, we present mDIA, the first large-scale multilingual benchmark for dialogue generation across low- to high-resource languages. It covers real-life conversations in 46 languages across 19 language families. We present baseline results obtained by fine-tuning the multilingual, non-dialogue-focused pre-trained model mT5 as well as English-centric, dialogue-focused pre-trained chatbot DialoGPT. The results show that mT5-based models perform better on sacreBLEU and BertScore but worse on diversity. Even though promising results are found in few-shot and zero-shot scenarios, there is a large gap between the generation quality in English and other languages. We hope that the release of mDIA could encourage more works on multilingual dialogue generation to promote language diversity.
@article{arxiv.2208.13078,
title = {MDIA: A Benchmark for Multilingual Dialogue Generation in 46 Languages},
author = {Qingyu Zhang and Xiaoyu Shen and Ernie Chang and Jidong Ge and Pengke Chen},
journal= {arXiv preprint arXiv:2208.13078},
year = {2022}
}
Comments
The dataset and processing scripts are available in https://github.com/DoctorDream/mDIA