On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering
Abstract
Visual Question Answering (VQA) methods have made incredible progress, but suffer from a failure to generalize. This is visible in the fact that they are vulnerable to learning coincidental correlations in the data rather than deeper relations between image content and ideas expressed in language. We present a dataset that takes a step towards addressing this problem in that it contains questions expressed in two languages, and an evaluation process that co-opts a well understood image-based metric to reflect the method's ability to reason. Measuring reasoning directly encourages generalization by penalizing answers that are coincidentally correct. The dataset reflects the scene-text version of the VQA problem, and the reasoning evaluation can be seen as a text-based version of a referring expression challenge. Experiments and analysis are provided that show the value of the dataset.
Keywords
Cite
@article{arxiv.2002.10215,
title = {On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answering},
author = {Xinyu Wang and Yuliang Liu and Chunhua Shen and Chun Chet Ng and Canjie Luo and Lianwen Jin and Chee Seng Chan and Anton van den Hengel and Liangwei Wang},
journal= {arXiv preprint arXiv:2002.10215},
year = {2020}
}
Comments
Accepted to Proc. IEEE Conf. Computer Vision and Pattern Recognition 2020