xSIM++: An Improved Proxy to Bitext Mining Performance for Low-Resource Languages
Abstract
We introduce a new proxy score for evaluating bitext mining based on similarity in a multilingual embedding space: xSIM++. In comparison to xSIM, this improved proxy leverages rule-based approaches to extend English sentences in any evaluation set with synthetic, hard-to-distinguish examples which more closely mirror the scenarios we encounter during large-scale mining. We validate this proxy by running a significant number of bitext mining experiments for a set of low-resource languages, and subsequently train NMT systems on the mined data. In comparison to xSIM, we show that xSIM++ is better correlated with the downstream BLEU scores of translation systems trained on mined bitexts, providing a reliable proxy of bitext mining performance without needing to run expensive bitext mining pipelines. xSIM++ also reports performance for different error types, offering more fine-grained feedback for model development.
Cite
@article{arxiv.2306.12907,
title = {xSIM++: An Improved Proxy to Bitext Mining Performance for Low-Resource Languages},
author = {Mingda Chen and Kevin Heffernan and Onur Çelebi and Alex Mourachko and Holger Schwenk},
journal= {arXiv preprint arXiv:2306.12907},
year = {2023}
}
Comments
The first two authors contributed equally; ACL 2023 short; Code and data are available at https://github.com/facebookresearch/LASER