InstrumentGen: Generating Sample-Based Musical Instruments From Text
Audio and Speech Processing
2023-11-09 v1 Machine Learning
Sound
Abstract
We introduce the text-to-instrument task, which aims at generating sample-based musical instruments based on textual prompts. Accordingly, we propose InstrumentGen, a model that extends a text-prompted generative audio framework to condition on instrument family, source type, pitch (across an 88-key spectrum), velocity, and a joint text/audio embedding. Furthermore, we present a differentiable loss function to evaluate the intra-instrument timbral consistency of sample-based instruments. Our results establish a foundational text-to-instrument baseline, extending research in the domain of automatic sample-based instrument generation.
Keywords
Cite
@article{arxiv.2311.04339,
title = {InstrumentGen: Generating Sample-Based Musical Instruments From Text},
author = {Shahan Nercessian and Johannes Imort},
journal= {arXiv preprint arXiv:2311.04339},
year = {2023}
}