A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understanding
Abstract
Potentially idiomatic expressions (PIEs) construe meanings inherently tied to the everyday experience of a given language community. As such, they constitute an interesting challenge for assessing the linguistic (and to some extent cultural) capabilities of NLP systems. In this paper, we present XMPIE, a parallel multilingual and multimodal dataset of potentially idiomatic expressions. The dataset, containing 34 languages and over ten thousand items, allows comparative analyses of idiomatic patterns among language-specific realisations and preferences in order to gather insights about shared cultural aspects. This parallel dataset allows to evaluate model performance for a given PIE in different languages and whether idiomatic understanding in one language can be transferred to another. Moreover, the dataset supports the study of PIEs across textual and visual modalities, to measure to what extent PIE understanding in one modality transfers or implies in understanding in another modality (text vs. image). The data was created by language experts, with both textual and visual components crafted under multilingual guidelines, and each PIE is accompanied by five images representing a spectrum from idiomatic to literal meanings, including semantically related and random distractors. The result is a high-quality benchmark for evaluating multilingual and multimodal idiomatic language understanding.
Cite
@article{arxiv.2601.08645,
title = {A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understanding},
author = {Dilara Torunoğlu-Selamet and Dogukan Arslan and Rodrigo Wilkens and Wei He and Doruk Eryiğit and Thomas Pickard and Adriana S. Pagano and Aline Villavicencio and Gülşen Eryiğit and Ágnes Abuczki and Aida Cardoso and Alesia Lazarenka and Dina Almassova and Amalia Mendes and Anna Kanellopoulou and Antoni Brosa-Rodríguez and Baiba Saulite and Beata Wojtowicz and Bolette Pedersen and Carlos Manuel Hidalgo-Ternero and Chaya Liebeskind and Danka Jokić and Diego Alves and Eleni Triantafyllidi and Erik Velldal and Fred Philippy and Giedre Valunaite Oleskeviciene and Ieva Rizgeliene and Inguna Skadina and Irina Lobzhanidze and Isabell Stinessen Haugen and Jauza Akbar Krito and Jelena M. Marković and Johanna Monti and Josue Alejandro Sauca and Kaja Dobrovoljc and Kingsley O. Ugwuanyi and Laura Rituma and Lilja Øvrelid and Maha Tufail Agro and Manzura Abjalova and Maria Chatzigrigoriou and María del Mar Sánchez Ramos and Marija Pendevska and Masoumeh Seyyedrezaei and Mehrnoush Shamsfard and Momina Ahsan and Muhammad Ahsan Riaz Khan and Nathalie Carmen Hau Norman and Nilay Erdem Ayyıldız and Nina Hosseini-Kivanani and Noémi Ligeti-Nagy and Numaan Naeem and Olha Kanishcheva and Olha Yatsyshyna and Daniil Orel and Petra Giommarelli and Petya Osenova and Radovan Garabik and Regina E. Semou and Rozane Rebechi and Salsabila Zahirah Pranida and Samia Touileb and Sanni Nimb and Sarfraz Ahmad and Sarvinoz Sharipova and Shahar Golan and Shaoxiong Ji and Sopuruchi Christian Aboh and Srdjan Sucur and Stella Markantonatou and Sussi Olsen and Vahide Tajalli and Veronika Lipp and Voula Giouli and Yelda Yeşildal Eraydın and Zahra Saaberi and Zhuohan Xie},
journal= {arXiv preprint arXiv:2601.08645},
year = {2026}
}