A recent trend in multimodal retrieval is related to postprocessing test set results via the dual-softmax loss (DSL). While this approach can bring significant improvements, it usually presumes that an entire matrix of test samples is available as DSL input. This work introduces a new postprocessing approach based on Sinkhorn transformations that outperforms DSL. Further, we propose a new postprocessing setting that does not require access to multiple test queries. We show that our approach can significantly improve the results of state of the art models such as CLIP4Clip, BLIP, X-CLIP, and DRL, thus achieving a new state-of-the-art on several standard text-video retrieval datasets both with access to the entire test set and in the single-query setting.
Cite
@article{arxiv.2311.08143,
title = {Sinkhorn Transformations for Single-Query Postprocessing in Text-Video Retrieval},
author = {Konstantin Yakovlev and Gregory Polyakov and Ilseyar Alimova and Alexander Podolskiy and Andrey Bout and Sergey Nikolenko and Irina Piontkovskaya},
journal= {arXiv preprint arXiv:2311.08143},
year = {2023}
}