Vis2Mus: Exploring Multimodal Representation Mapping for Controllable Music Generation
Abstract
In this study, we explore the representation mapping from the domain of visual arts to the domain of music, with which we can use visual arts as an effective handle to control music generation. Unlike most studies in multimodal representation learning that are purely data-driven, we adopt an analysis-by-synthesis approach that combines deep music representation learning with user studies. Such an approach enables us to discover \textit{interpretable} representation mapping without a huge amount of paired data. In particular, we discover that visual-to-music mapping has a nice property similar to equivariant. In other words, we can use various image transformations, say, changing brightness, changing contrast, style transfer, to control the corresponding transformations in the music domain. In addition, we released the Vis2Mus system as a controllable interface for symbolic music generation.
Cite
@article{arxiv.2211.05543,
title = {Vis2Mus: Exploring Multimodal Representation Mapping for Controllable Music Generation},
author = {Runbang Zhang and Yixiao Zhang and Kai Shao and Ying Shan and Gus Xia},
journal= {arXiv preprint arXiv:2211.05543},
year = {2022}
}
Comments
Submitted to ICASSP 2023. GitHub repo: https://github.com/ldzhangyx/vis2mus