Self-Supervised Learning for Visual Relationship Detection through Masked Bounding Box Reconstruction
Abstract
We present a novel self-supervised approach for representation learning, particularly for the task of Visual Relationship Detection (VRD). Motivated by the effectiveness of Masked Image Modeling (MIM), we propose Masked Bounding Box Reconstruction (MBBR), a variation of MIM where a percentage of the entities/objects within a scene are masked and subsequently reconstructed based on the unmasked objects. The core idea is that, through object-level masked modeling, the network learns context-aware representations that capture the interaction of objects within a scene and thus are highly predictive of visual object relationships. We extensively evaluate learned representations, both qualitatively and quantitatively, in a few-shot setting and demonstrate the efficacy of MBBR for learning robust visual representations, particularly tailored for VRD. The proposed method is able to surpass state-of-the-art VRD methods on the Predicate Detection (PredDet) evaluation setting, using only a few annotated samples. We make our code available at https://github.com/deeplab-ai/SelfSupervisedVRD.
Cite
@article{arxiv.2311.04834,
title = {Self-Supervised Learning for Visual Relationship Detection through Masked Bounding Box Reconstruction},
author = {Zacharias Anastasakis and Dimitrios Mallis and Markos Diomataris and George Alexandridis and Stefanos Kollias and Vassilis Pitsikalis},
journal= {arXiv preprint arXiv:2311.04834},
year = {2023}
}
Comments
Camera Ready paper version of WACV 2024