Modeling Context in Referring Expressions
Abstract
Humans refer to objects in their environments all the time, especially in dialogue with other people. We explore generating and comprehending natural language referring expressions for objects in images. In particular, we focus on incorporating better measures of visual context into referring expression models and find that visual comparison to other objects within an image helps improve performance significantly. We also develop methods to tie the language generation process together, so that we generate expressions for all objects of a particular category jointly. Evaluation on three recent datasets - RefCOCO, RefCOCO+, and RefCOCOg, shows the advantages of our methods for both referring expression generation and comprehension.
Cite
@article{arxiv.1608.00272,
title = {Modeling Context in Referring Expressions},
author = {Licheng Yu and Patrick Poirson and Shan Yang and Alexander C. Berg and Tamara L. Berg},
journal= {arXiv preprint arXiv:1608.00272},
year = {2016}
}
Comments
19 pages, 6 figures, in ECCV 2016; authors, references and acknowledgement updated