We propose Unified Visual-Semantic Embeddings (UniVSE) for learning a joint space of visual and textual concepts. The space unifies the concepts at different levels, including objects, attributes, relations, and full scenes. A contrastive learning approach is proposed for the fine-grained alignment from only image-caption pairs. Moreover, we present an effective approach for enforcing the coverage of semantic components that appear in the sentence. We demonstrate the robustness of Unified VSE in defending text-domain adversarial attacks on cross-modal retrieval tasks. Such robustness also empowers the use of visual cues to resolve word dependencies in novel sentences.
@article{arxiv.1904.05521,
title = {UniVSE: Robust Visual Semantic Embeddings via Structured Semantic Representations},
author = {Hao Wu and Jiayuan Mao and Yufeng Zhang and Yuning Jiang and Lei Li and Weiwei Sun and Wei-Ying Ma},
journal= {arXiv preprint arXiv:1904.05521},
year = {2019}
}
Comments
v1 is the full version which is accepted by CVPR 2019. v2 is the short version accepted by NAACL 2019 SpLU-RoboNLP workshop (in non-archival proceedings)