Panoptic narrative grounding (PNG) aims to segment things and stuff objects in an image described by noun phrases of a narrative caption. As a multimodal task, an essential aspect of PNG is the visual-linguistic interaction between image and caption. The previous two-stage method aggregates visual contexts from offline-generated mask proposals to phrase features, which tend to be noisy and fragmentary. The recent one-stage method aggregates only pixel contexts from image features to phrase features, which may incur semantic misalignment due to lacking object priors. To realize more comprehensive visual-linguistic interaction, we propose to enrich phrases with coupled pixel and object contexts by designing a Phrase-Pixel-Object Transformer Decoder (PPO-TD), where both fine-grained part details and coarse-grained entity clues are aggregated to phrase features. In addition, we also propose a PhraseObject Contrastive Loss (POCL) to pull closer the matched phrase-object pairs and push away unmatched ones for aggregating more precise object contexts from more phrase-relevant object tokens. Extensive experiments on the PNG benchmark show our method achieves new state-of-the-art performance with large margins.
@article{arxiv.2311.01091,
title = {Enriching Phrases with Coupled Pixel and Object Contexts for Panoptic Narrative Grounding},
author = {Tianrui Hui and Zihan Ding and Junshi Huang and Xiaoming Wei and Xiaolin Wei and Jiao Dai and Jizhong Han and Si Liu},
journal= {arXiv preprint arXiv:2311.01091},
year = {2024}
}
Comments
Accepted by IJCAI 2023. Since the PNG benchmark adopts a different data partition manner from ours, we update the experimental results on the things/stuff/singulars/plurals subsets based on the PNG's code