MAAS: Multi-modal Assignation for Active Speaker Detection
Abstract
Active speaker detection requires a solid integration of multi-modal cues. While individual modalities can approximate a solution, accurate predictions can only be achieved by explicitly fusing the audio and visual features and modeling their temporal progression. Despite its inherent muti-modal nature, current methods still focus on modeling and fusing short-term audiovisual features for individual speakers, often at frame level. In this paper we present a novel approach to active speaker detection that directly addresses the multi-modal nature of the problem, and provides a straightforward strategy where independent visual features from potential speakers in the scene are assigned to a previously detected speech event. Our experiments show that, an small graph data structure built from a single frame, allows to approximate an instantaneous audio-visual assignment problem. Moreover, the temporal extension of this initial graph achieves a new state-of-the-art on the AVA-ActiveSpeaker dataset with a mAP of 88.8\%.
Cite
@article{arxiv.2101.03682,
title = {MAAS: Multi-modal Assignation for Active Speaker Detection},
author = {Juan León-Alcázar and Fabian Caba Heilbron and Ali Thabet and Bernard Ghanem},
journal= {arXiv preprint arXiv:2101.03682},
year = {2021}
}