A comprehensive representation of an image requires understanding objects and their mutual relationship, especially in image-to-graph generation, e.g., road network extraction, blood-vessel network extraction, or scene graph generation. Traditionally, image-to-graph generation is addressed with a two-stage approach consisting of object detection followed by a separate relation prediction, which prevents simultaneous object-relation interaction. This work proposes a unified one-stage transformer-based framework, namely Relationformer, that jointly predicts objects and their relations. We leverage direct set-based object prediction and incorporate the interaction among the objects to learn an object-relation representation jointly. In addition to existing [obj]-tokens, we propose a novel learnable token, namely [rln]-token. Together with [obj]-tokens, [rln]-token exploits local and global semantic reasoning in an image through a series of mutual associations. In combination with the pair-wise [obj]-token, the [rln]-token contributes to a computationally efficient relation prediction. We achieve state-of-the-art performance on multiple, diverse and multi-domain datasets that demonstrate our approach's effectiveness and generalizability.
@article{arxiv.2203.10202,
title = {Relationformer: A Unified Framework for Image-to-Graph Generation},
author = {Suprosanna Shit and Rajat Koner and Bastian Wittmann and Johannes Paetzold and Ivan Ezhov and Hongwei Li and Jiazhen Pan and Sahand Sharifzadeh and Georgios Kaissis and Volker Tresp and Bjoern Menze},
journal= {arXiv preprint arXiv:2203.10202},
year = {2022}
}