Automatically describing audio-visual content with texts, namely video captioning, has received significant attention due to its potential applications across diverse fields. Deep neural networks are the dominant methods, offering state-of-the-art performance. However, these methods are often undeployable in low-power devices like smartphones due to the large size of the model parameters. In this paper, we propose to exploit simple pooling front-end and down-sampling algorithms with knowledge distillation for audio and visual attributes using a reduced number of audio-visual frames. With the help of knowledge distillation from the teacher model, our proposed method greatly reduces the redundant information in audio-visual streams without losing critical contexts for caption generation. Extensive experimental evaluations on the MSR-VTT dataset demonstrate that our proposed approach significantly reduces the inference time by about 80% with a small sacrifice (less than 0.02%) in captioning accuracy.
@article{arxiv.2306.09947,
title = {Knowledge Distillation for Efficient Audio-Visual Video Captioning},
author = {Özkan Çaylı and Xubo Liu and Volkan Kılıç and Wenwu Wang},
journal= {arXiv preprint arXiv:2306.09947},
year = {2023}
}
Comments
European Signal Processing Conference (EUSIPCO 2023)