Music-to-Text Synaesthesia: Generating Descriptive Text from Music Recordings
Abstract
In this paper, we consider a novel research problem: music-to-text synaesthesia. Different from the classical music tagging problem that classifies a music recording into pre-defined categories, music-to-text synaesthesia aims to generate descriptive texts from music recordings with the same sentiment for further understanding. As existing music-related datasets do not contain the semantic descriptions on music recordings, we collect a new dataset that contains 1,955 aligned pairs of classical music recordings and text descriptions. Based on this, we build a computational model to generate sentences that can describe the content of the music recording. To tackle the highly non-discriminative classical music, we design a group topology-preservation loss, which considers more samples as a group reference and preserves the relative topology among different samples. Extensive experimental results qualitatively and quantitatively demonstrate the effectiveness of our proposed model over five heuristics or pre-trained competitive methods and their variants on our collected dataset.
Keywords
Cite
@article{arxiv.2210.00434,
title = {Music-to-Text Synaesthesia: Generating Descriptive Text from Music Recordings},
author = {Zhihuan Kuang and Shi Zong and Jianbing Zhang and Jiajun Chen and Hongfu Liu},
journal= {arXiv preprint arXiv:2210.00434},
year = {2023}
}