The link prediction task on knowledge graphs without explicit negative triples in the training data motivates the usage of rank-based metrics. Here, we review existing rank-based metrics and propose desiderata for improved metrics to address lack of interpretability and comparability of existing metrics to datasets of different sizes and properties. We introduce a simple theoretical framework for rank-based metrics upon which we investigate two avenues for improvements to existing metrics via alternative aggregation functions and concepts from probability theory. We finally propose several new rank-based metrics that are more easily interpreted and compared accompanied by a demonstration of their usage in a benchmarking of knowledge graph embedding models.
@article{arxiv.2203.07544,
title = {A Unified Framework for Rank-based Evaluation Metrics for Link Prediction in Knowledge Graphs},
author = {Charles Tapley Hoyt and Max Berrendorf and Mikhail Galkin and Volker Tresp and Benjamin M. Gyori},
journal= {arXiv preprint arXiv:2203.07544},
year = {2022}
}
Comments
Accepted at the Workshop on Graph Learning Benchmarks @ The WebConf 2022