Quantifying the amount of visual information used by neural caption generators
Neural and Evolutionary Computing
2019-02-05 v1 Computation and Language
Abstract
This paper addresses the sensitivity of neural image caption generators to their visual input. A sensitivity analysis and omission analysis based on image foils is reported, showing that the extent to which image captioning architectures retain and are sensitive to visual information varies depending on the type of word being generated and the position in the caption as a whole. We motivate this work in the context of broader goals in the field to achieve more explainability in AI.
Keywords
Cite
@article{arxiv.1810.05475,
title = {Quantifying the amount of visual information used by neural caption generators},
author = {Marc Tanti and Albert Gatt and Kenneth P. Camilleri},
journal= {arXiv preprint arXiv:1810.05475},
year = {2019}
}
Comments
10 pages, 4 figures This publication will appear in the Proceedings of the First Workshop on Shortcomings in Vision and Language (2018). DOI to be inserted later