Many previous methods have demonstrated the importance of considering semantically relevant objects for carrying out video-based human activity recognition, yet none of the methods have harvested the power of large text corpora to relate the objects and the activities to be transferred into learning a unified deep convolutional neural network. We present a novel activity recognition CNN which co-learns the object recognition task in an end-to-end multitask learning scheme to improve upon the baseline activity recognition performance. We further improve upon the multitask learning approach by exploiting a text-guided semantic space to select the most relevant objects with respect to the target activities. To the best of our knowledge, we are the first to investigate this approach.
@article{arxiv.1805.01818,
title = {Object and Text-guided Semantics for CNN-based Activity Recognition},
author = {Sungmin Eum and Christopher Reale and Heesung Kwon and Claire Bonial and Clare Voss},
journal= {arXiv preprint arXiv:1805.01818},
year = {2018}
}