Measuring the Measuring Tools: An Automatic Evaluation of Semantic Metrics for Text Corpora
Abstract
The ability to compare the semantic similarity between text corpora is important in a variety of natural language processing applications. However, standard methods for evaluating these metrics have yet to be established. We propose a set of automatic and interpretable measures for assessing the characteristics of corpus-level semantic similarity metrics, allowing sensible comparison of their behavior. We demonstrate the effectiveness of our evaluation measures in capturing fundamental characteristics by evaluating them on a collection of classical and state-of-the-art metrics. Our measures revealed that recently-developed metrics are becoming better in identifying semantic distributional mismatch while classical metrics are more sensitive to perturbations in the surface text levels.
Keywords
Cite
@article{arxiv.2211.16259,
title = {Measuring the Measuring Tools: An Automatic Evaluation of Semantic Metrics for Text Corpora},
author = {George Kour and Samuel Ackerman and Orna Raz and Eitan Farchi and Boaz Carmeli and Ateret Anaby-Tavor},
journal= {arXiv preprint arXiv:2211.16259},
year = {2022}
}
Comments
Published at GEM (https://gem-benchmark.com/workshop) workshop at the Empirical Methods in Natural Language Processing (EMNLP) conference in 2022