Cascaded Cross-Modal Transformer for Request and Complaint Detection
Computation and Language
2023-07-31 v1 Machine Learning
Multimedia
Audio and Speech Processing
Abstract
We propose a novel cascaded cross-modal transformer (CCMT) that combines speech and text transcripts to detect customer requests and complaints in phone conversations. Our approach leverages a multimodal paradigm by transcribing the speech using automatic speech recognition (ASR) models and translating the transcripts into different languages. Subsequently, we combine language-specific BERT-based models with Wav2Vec2.0 audio features in a novel cascaded cross-attention transformer model. We apply our system to the Requests Sub-Challenge of the ACM Multimedia 2023 Computational Paralinguistics Challenge, reaching unweighted average recalls (UAR) of 65.41% and 85.87% for the complaint and request classes, respectively.
Cite
@article{arxiv.2307.15097,
title = {Cascaded Cross-Modal Transformer for Request and Complaint Detection},
author = {Nicolae-Catalin Ristea and Radu Tudor Ionescu},
journal= {arXiv preprint arXiv:2307.15097},
year = {2023}
}
Comments
Accepted at ACMMM 2023