Sonification of Facial Actions for Musical Expression
Human-Computer Interaction
2020-10-08 v1 Computer Vision and Pattern Recognition
Sound
Audio and Speech Processing
Abstract
The central role of the face in social interaction and non-verbal communication suggests we explore facial action as a means of musical expression. This paper presents the design, implementation, and preliminary studies of a novel system utilizing face detection and optic flow algorithms to associate facial movements with sound synthesis in a topographically specific fashion. We report on our experience with various gesture-to-sound mappings and applications, and describe our preliminary experiments at musical performance using the system.
Keywords
Cite
@article{arxiv.2010.03223,
title = {Sonification of Facial Actions for Musical Expression},
author = {Mathias Funk and Kazuhiro Kuwabara and Michael J. Lyons},
journal= {arXiv preprint arXiv:2010.03223},
year = {2020}
}
Comments
Proceedings of the International Conference on New Interfaces for Musical Expression, 2005 (NIME-05)