Measuring Machine Intelligence Through Visual Question Answering
Abstract
As machines have become more intelligent, there has been a renewed interest in methods for measuring their intelligence. A common approach is to propose tasks for which a human excels, but one which machines find difficult. However, an ideal task should also be easy to evaluate and not be easily gameable. We begin with a case study exploring the recently popular task of image captioning and its limitations as a task for measuring machine intelligence. An alternative and more promising task is Visual Question Answering that tests a machine's ability to reason about language and vision. We describe a dataset unprecedented in size created for the task that contains over 760,000 human generated questions about images. Using around 10 million human generated answers, machines may be easily evaluated.
Cite
@article{arxiv.1608.08716,
title = {Measuring Machine Intelligence Through Visual Question Answering},
author = {C. Lawrence Zitnick and Aishwarya Agrawal and Stanislaw Antol and Margaret Mitchell and Dhruv Batra and Devi Parikh},
journal= {arXiv preprint arXiv:1608.08716},
year = {2016}
}
Comments
AI Magazine, 2016