Translating Visual Art into Music
Computer Vision and Pattern Recognition
2019-09-15 v1 Human-Computer Interaction
Machine Learning
Sound
Audio and Speech Processing
Abstract
The Synesthetic Variational Autoencoder (SynVAE) introduced in this research is able to learn a consistent mapping between visual and auditive sensory modalities in the absence of paired datasets. A quantitative evaluation on MNIST as well as the Behance Artistic Media dataset (BAM) shows that SynVAE is capable of retaining sufficient information content during the translation while maintaining cross-modal latent space consistency. In a qualitative evaluation trial, human evaluators were furthermore able to match musical samples with the images which generated them with accuracies of up to 73%.
Cite
@article{arxiv.1909.01218,
title = {Translating Visual Art into Music},
author = {Maximilian Müller-Eberstein and Nanne van Noord},
journal= {arXiv preprint arXiv:1909.01218},
year = {2019}
}
Comments
Accepted for ICCV 2019 Workshop on Fashion, Art and Design