Evaluating image captions without references remains challenging because global embedding similarity often misses fine-grained mismatches such as hallucinated objects, missing attributes, or incorrect relations. We propose MSD-Score, a reference-free metric that models image patch and text token embeddings as von Mises-Fisher mixtures on the unit hypersphere. Instead of treating each modality as a single point, MSD-Score formulates image-text matching as a multi-scale distributional scoring problem. Semantic discrepancies are quantified via a weighted bi-directional KL divergence and combined with global similarity in a multi-scale framework for both single- and multi-candidate evaluations. Extensive experiments show that MSD-Score achieves state-of-the-art correlation with human judgments among reference-free metrics. Beyond accuracy, its probabilistic formulation yields transparent and decomposable diagnostics of local grounding errors, providing a deterministic complementary signal to holistic similarity metrics and judge-based evaluators.
@article{arxiv.2605.06080,
title = {MSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluation},
author = {Shichao Kan and Xuyang Zhang and Haojie Zhang and Zhe Zhu and Yigang Cen and Yixiong Liang and Lianlei Shan and Linna Zhang and Zhe Qu and Jiazhi Xia},
journal= {arXiv preprint arXiv:2605.06080},
year = {2026}
}
Comments
Preprint. 17 pages, 10 figures. Code is available at: https://steinsgatesg.github.io/MSDScore/