Semantic Search Evaluation
Information Retrieval
2024-10-30 v1 Computation and Language
Abstract
We propose a novel method for evaluating the performance of a content search system that measures the semantic match between a query and the results returned by the search system. We introduce a metric called "on-topic rate" to measure the percentage of results that are relevant to the query. To achieve this, we design a pipeline that defines a golden query set, retrieves the top K results for each query, and sends calls to GPT 3.5 with formulated prompts. Our semantic evaluation pipeline helps identify common failure patterns and goals against the metric for relevance improvements.
Cite
@article{arxiv.2410.21549,
title = {Semantic Search Evaluation},
author = {Chujie Zheng and Jeffrey Wang and Shuqian Albee Zhang and Anand Kishore and Siddharth Singh},
journal= {arXiv preprint arXiv:2410.21549},
year = {2024}
}
Comments
Accepted by 3rd International Workshop on Industrial Recommendation Systems (at CIKM 2024)