'Tis but Thy Name: Semantic Question Answering Evaluation with 11M Names for 1M Entities

Albert Huang

'Tis but Thy Name: Semantic Question Answering Evaluation with 11M Names for 1M Entities

Computation and Language 2022-03-01 v1

Authors: Albert Huang

Abstract

Classic lexical-matching-based QA metrics are slowly being phased out because they punish succinct or informative outputs just because those answers were not provided as ground truth. Recently proposed neural metrics can evaluate semantic similarity but were trained on small textual similarity datasets grafted from foreign domains. We introduce the Wiki Entity Similarity (WES) dataset, an 11M example, domain targeted, semantic entity similarity dataset that is generated from link texts in Wikipedia. WES is tailored to QA evaluation: the examples are entities and phrases and grouped into semantic clusters to simulate multiple ground-truth labels. Human annotators consistently agree with WES labels, and a basic cross encoder metric is better than four classic metrics at predicting human judgments of correctness.

Keywords

question answering evaluation semantic similarity wiki entity similarity entity clusters cross encoder human judgment dataset generation

Cite

@article{arxiv.2202.13581,
  title  = {'Tis but Thy Name: Semantic Question Answering Evaluation with 11M Names for 1M Entities},
  author = {Albert Huang},
  journal= {arXiv preprint arXiv:2202.13581},
  year   = {2022}
}

Comments

4 pages, 2 figures. Will be updated with more analysis and baseline models

'Tis but Thy Name: Semantic Question Answering Evaluation with 11M Names for 1M Entities

Abstract

Keywords

Cite

Comments

Related papers