This paper investigates how linguistic knowledge mined from large text corpora can aid the generation of natural language descriptions of videos. Specifically, we integrate both a neural language model and distributional semantics trained on large text corpora into a recent LSTM-based architecture for video description. We evaluate our approach on a collection of Youtube videos as well as two large movie description datasets showing significant improvements in grammaticality while modestly improving descriptive quality.
@article{arxiv.1604.01729,
title = {Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text},
author = {Subhashini Venugopalan and Lisa Anne Hendricks and Raymond Mooney and Kate Saenko},
journal= {arXiv preprint arXiv:1604.01729},
year = {2016}
}
Comments
Accepted at EMNLP 2016. Project page: http://vsubhashini.github.io/language_fusion.html